Visual Basic 6 and .NET COM Interop

This is a quite popular topic on the Internet, and this has a reason: a lot of developers need to integrate legacy components written in Visual Basic 6 with .NET.
COM Interop is or should be one of the easiest ways to bring these two worlds together.

The problem begins when multiple threads, apartments or custom data types float into the picture.

Today I’d like to share an easy way to tackle with data type interop, plus an unexplained discovery I made during a recent case.

The problem

My customer had a .NET method defined as follows:

public void SetData(Data[] param)

{

...

}

‘Data’ was his custom type, also defined in .NET.

When calling this method from Visual Basic 6 through COM interop, we get a run-time error: “Function or interface marked as restricted, or the function uses an Automation type not supported in Visual Basic”.

It’s all about the TLB

When data is passed from client to server or back, chances are that it must be marshaled (i.e. converted). There’re a number of conditions that dictate this, e.g. different representation of data, the need to serialize and transmit data over network, etc.

However, (automated) marshaling cannot happen without some metadata describing the actual data types that are involved. With the advent of .NET, metadata is stored right in the assembly with the code. But in terms of COM, it was stored in a Type Library (TLB). The TLB itself is just a binary representation of the Interface Definition Language (IDL). TLB/IDL is the “common language” that all COM components or clients speak. It’s worth noting that IDL is a powerful language that can express constructs which are not supported by every programming language or runtime. Also keep in mind that a TLB can be included within an EXE or a DLL as a resource.

As for Visual Basic 6, it also relies on the TLB to make the types of the component usable in the client. When doing COM interop, the TLB is generated by the .NET SDK tools tlbexp.exe or regasm.exe.

Therefore, if using a managed type via COM interop from Visual Basic 6 doesn’t work out-of-the-box, then very likely there’s a problem with the TLB/IDL. The problem is not that the TLB is invalid, since then it couldn’t have been compiled at all. Rather, the most likely issue here is that one party (e.g. .NET) expresses a type with a construct that is not supported by the other party (Visual Basic 6). Some constructs do not exist at all, while others are just expressed otherwise. In the latter case, often the difference is not just syntactical, and thus the run-time of Visual Basic 6, for example, won’t be able to handle it.

Thus, we need to look at the TLB to find the problem. Simply look up the type that you’re trying to use, and see how it was defined.

Tool support

Luckily, we don’t need to read TLBs in binary. There’s a tool included with the Windows SDK, called oleview.exe. Just pass the TLB as a command-line parameter and it will show the IDL. It will also allow passing EXE or DLL files, as long as they contain a TLB.

Finding out what’s wrong

In our case, the relevant method looked like:

[id(0x60020020)]
HRESULT SetData([in] SAFEARRAY(_Data*) param);

More often than not, we don’t have an idea about why Visual Basic 6 doesn’t like a given construct. An easy way to tell this is to create a simple ActiveX project in Visual Basic 6 and define a method or type that resembles the construct on the .NET side the most. Then, compile the project, pass the DLL/EXE to oleview.exe and check out how Visual Basic 6 expressed what you wrote in IDL.

We did just the same, and what we saw was:

[id(0x60030000)]
HRESULT SetData([in, out] SAFEARRAY(_Data*)* param);

The difference was an additional level of indirection, highlighted above.

System.Runtime.InteropServices.MarshalAsAttribute comes at the rescue

Since you have close to zero ways to customize things on the Visual Basic 6 size, you need to look at the managed code. Since now you know how the managed type or method should appear to COM consumers, you can change the .NET definition to accommodate this. Fortunately, this attribute allows to change the IDL (and the behavior of the .NET marshaler) without changing the .NET semantics. It goes far beyond the scope of this post to describe what this attribute can do, you can have a look at MSDN.

Sometimes, however, you do need to change .NET semantics, like in our case. Since we need an additional level of indirection, we need to pass the array by reference – thus use the ‘ref’ keyword:

public void SetData(ref Data[] param)

{

...

}

Bonus tip

During this recent support case with my customer, we proceeded up to this point, but still Visual Basic 6 threw run-time errors when using the modified method. After a lot of trial and errors, we discovered that invoking the method as a procedure (i.e. without parentheses) was always successful, while the function-like invoking style (i.e. with parentheses) always failed.

Seasoned Visual Basic 6 developers may be able to explain this, I could not. If anyone has an idea, feel free to add it to comments.

Comments

• Anonymous
  November 13, 2009
  here's the problem:  VB6 app using .Net CustomControl via COM Interop (MS Interop Toolkit). When registered with regasm on one machine - everything works fine. WHen registered the same way on other, getting mysterious "Out of memory" error message and nothing logged in event log... why??? this has been killing me :(  both machines have same set of .Net frameworks installed from 1.0 up to 3.5... everything seems the same, yet behaves differently...   cannot use reg-free com on either machine, always getting "out of memory"... any ideas???

• Anonymous
  November 16, 2009
  Ray, this out of memory error is very likely misleading, and thus it's hard to say what's causing the problem.Here're a few ideas:Try not using registration-free COM (although regasm indicates that actually you don't use it) Use Fusion Log Viewer to detect assembly-loading problems If all else fails, use Process Monitor to see if all registry entries / files are not found and can be accessed.

• Anonymous
  March 31, 2010
  Thanks.  Had a problem with a .NET dll created for COM use, being used by Excel VBA where one of the method parameters was a string array. Got the "Function or interface marked as restricted..." error in Excel when trying to use the method. Solution was to make that parameter a 'ref' parameter in the .NET library so it was called by reference and that allowed things to work.

• Anonymous
  September 03, 2010
  Not only Sub but also Function call works as long as the return variable is being handled eg :   Public Function Concat1(ByRef mystring() As String) As Boolean Implements _demo.RecCC       Dim i As Integer = 0       For i = 0 To UBound(mystring)           ReCC &= mystring(i) & ","       Next       ReCC = Mid(ReCC, 1, ReCC.Length - 1)       RecCC = ReCC   End FunctionCalling the same in vb6Dim blnmytest As Booleanblnmytest = cClass.RecCC(strRecCC())

• Anonymous
  December 06, 2011
  I was trying to pass arrays of dates and structures from VB6 to C# via COM interop and was also getting the "Function or interface marked as restricted..." error in the VB6 IDE. Adding "ref" to my array parameters in the C# code fixed it for me too.