Oil and water - MSMQ transactional messages and load balancing

A really common request is to load-balance MSMQ, either to "scale out" (adding more machines, as opposed to "scale up" by enhancing the existing server) or for high availability.

For load-balancing to work, the traffic must be stateless as there is no guarantee that a sending machine will be using the same balancer each time.

For express messages there isn't a problem as these are stateless - "fire and forget".

Transactional messages, though, are not going to be happy with load balancing systems. The message transfer protocol for transactional messaging needs state consistency between the sender and the receiver (state is maintained at both ends) and hence it becomes tricky to achieve exactly-once-in-order delivery guarantee in a stateless load-balancing farm or a configuration that involves a network address translation in the middle. Also, transactional messaging requires messages to flow in both directions so ports need to be open in the outward direction as well.

The most common symptom is that storage acknowledgements returning to the sender are sent to the wrong machine. This is because the acknowledgement message is sent to the IP address contained in the message it is acknowledging. Unfortunately that is the IP address of the load balancing machine. The result is that the load balancer receives the acknowledgement but has no way of working out which machine to forward it on to. The wrong machine will simply discard the acknowledgement. Without the acknowledgment, the sending machine cannot confirm message delivery so messages just accumulate in the outgoing queue.

The subject is discussed in more detail here:

899611 How Message Queuing can function over Network Load Balancing (NLB).

There are a couple of workarounds (including one too new for inclusion in the KB):

• Using a store-and-forward server
  Read the "Sending Message Queuing Messages in a Network Load Balancing Farm Configuration" section of the Microsoft Message Queuing (MSMQ) HTTP Deployment Scenarios for Windows Server 2003 and Windows XP Professional white paper.
• Using remote transactional reads (MSMQ 4.0 - Windows Vista and Windows Server 2008).
  Messages are stored on a central server but you scale out the processing servers which read remotely from it.

[[Thanks for Sanjib Saha for ideas]]

Comments

• Anonymous
  January 06, 2010
  "Transactional messages, though, are not going to be happy with load balancing systems. The message transfer protocol for transactional messaging needs state consistency between the sender and the receiver (state is maintained at both ends)" I hadn't realized this was the case, are you aware of any content regarding this that I could read to get more info?

• Anonymous
  January 06, 2010
  Hi Colin, I'm not sure if there's something really detailed - I'm going on what I learn in my support role. Any document that talks about MQInSeqs.lgx and MQTrans.lgx files is a good place to start. For example: Storage Folder Files http://msdn.microsoft.com/en-us/library/ms704178(VS.85).aspx "The files in the MSMQStorage folder include ... state files for received message (MQInSeqs.lgx files) and transaction operations (MQTrans.lgx files)." This state information cannot be shared between all the MSMQ nodes.