Really large messages with MSMQT and MSMQ (MSMQ/C) Adapter

When it comes to really large messages – and I don’t mean merely 50Mb – there is a difference between the old MSMQT adapter and the new MSMQ (a.k.a. MSMQ/C) Adapter for BizTalk.

 

The new MSMQ Adapter is using MQRTLarge.dll that was shipped originally with BizTalk 2004 to enable communicating large messages (exceeding standard 4Mb limit of MSMQ) to BizTalk. It works like this: you put MQRTLarge.dll on the machine with regular MSMQ, and you change your application to work with MQRTLarge API instead of the regular MSMQ API. They are very much alike, almost identical, except that you link against a different DLL and now you magically can pass messages more than 4Mb. Everything else is done under the covers. The message is broken into parts,  the parts are delivered in order, and assembled on another end. Physically, you can even make it work between two MSMQ-only machines without BizTalk. It’s just not an officially supported way, because we shipped MQRTLarge just to enable communicating with large messages with BizTalk, that’s it.

 

There is one caveat though. Because MQRTLarge.dll API is basically the same as MSMQ API, you have to accumulate the whole message in memory, and if you want a very large message (say, 500Mb), you need a lot of memory. That puts effective limits on the size of the message you can send with MQRTLarge.dll. It’s a way higher than the old one of 4Mb and it can be improved even more with an extra memory, but it is still there.

 

So, if you use the new MSMQ Adapter (MSMQ/C), which is based on general MSMQ and MQRTLarge, you get this limit. Again, it’s really high, I successfully passed 100Mb messages on a machine with less than 800Mb of memory, but the limit is there.

 

In contrast to that, with MSMQT all message processing is streamed, so technically there is no limit. Of course, we put a maximum supported size into the docs (somewhere around 200Mb), which simply means that we extensively tested our ability to work with messages up to this size and hence we feel that this is what we can provide the support for. However, technically, we were able to pass 1Gb messages through MSMQT.

 

Now, an interesting question: do we really need 1Gb messages today?

 

As I mentioned above, it’s not “officially supported”, but people do non-supported things when they see them work. On another hand, a system that passes 1Gb messages will need a serious bandwidth, huge storage and a lot of processing power. See for yourself, an average small server with 200-300 Gb disk will be able to hold about a hundred of such messages or less (keep in mind: if it is supposed to really process them, extra copies will be created.) That’s not too much.

Also, what could you possibly have of such size? In most cases, it looks like a really wasteful design to send such large messages. Of course, I can imagine movies routed to subscribers this way, but why would you use BizTalk (or any other integration product) for such routing – there is no data conversion, right? Similarly, why would you use MSMQ? MSMQ as any asynchronous transport is designed to primarily decouple send and receive for unreliable networks. If you send a high-definition movie, you need a reliable network.

 

Another scenario I’ve heard of, is moving the whole project documentation through the system as a huge zip-file. This also does not sound like a great idea, because the same thing may be done with SharePoint Server holding the documentation, and only small work items (like request, review, approval) floating through the system.

 

So I wonder, if anybody really uses BizTalk for processing of 1Gb messages? If you do, can you let me know?