MSMQ's Major Moving Parts, part one

In MSMQ, there are two important kinds of entities. As you might expect, they are messages and queues.

To MSMQ, a message is a bundle of properties. The thing you normally think of as being the message, the actual content, is one of those properties, known as the body. A survey of message properties would be an entire post by itself, so I'm going to leave that for later. Every message in the system is required to be associated with ("in") a queue.

A queue is a logical storage location which can have messages associated with it. I call them logical because the physical storage of messages, whether in memory or on disk, is handled without regard to what queue the message is in. Unlike messages, which must be in a queue, a queue may contain zero messages and frequently does. Queues may be logical rather than physical, but each queue is "hosted" on exactly one machine. In other words, all messages in a particular queue are physically stored on the machine where the queue is hosted — they aren't non-physical enough to span multiple machines!

How does a message get from sender to receiver? It starts when an application opens a queue for sending, getting a handle back. That action indicates that the queue named in the call will be the destination for messages sent using that handle. The queue may be hosted on the same machine that the application is running on ("local") or on some other machine ("remote"). If the desired destination queue is local, then MSMQ checks to see whether it exists and whether the application has permission to send to it before allowing the open to succeed. If the desired destination is remote, MSMQ performs no checks; to be asynchronous, MSMQ must allow the opening of queues hosted on machines which are not currently reachable, and that was most easily achieved by having the destination machine do those checks when the messages finally arrive there. But I'm getting ahead of myself!

Once the application has a handle to a queue, it assembles a message and then makes an API call to send it. Messages sent to local queues are simply placed directly in their destination. Messages sent to remote queues require more complicated handling. Every message in the system must be in a queue, so MSMQ must find a queue to put those messages in until they can be delivered across the network to the destination machine. These are called "outgoing queues" and they are created as needed and cleaned up when they have been empty for more than a certain amount of time. MSMQ's MMC snapin allows administrators to view outgoing queues, which can provide valuable information about message flow between two machines.

MSMQ periodically tries to connect to each of the machines for which it has messages waiting in an outgoing queue. When a connection is successfully established, it transfers the messages across the network. MSMQ on the destination machine performs the checks which were not performed earlier, since the destination queue is local there, and the appropriate error action is taken if the queue does not exist or the sender does not have the required permissions. Because MSMQ is asynchronous, the sending application's API calls to open the queue and send the message are long since completed — in fact, the sending application may not even be running anymore! — so there is no way to return an error code. The error action typically consists of sending back a negative acknowledgement (another kind of message) or placing the message in a deadletter queue (more on those in another post). If all the checks pass, then the destination machine places the message in the destination queue, where it can wait for another application to receive it. Messages can wait to be received indefinitely, depending on various system-wide and message-specific settings.

An application receives a message by opening a queue for receiving. Like sending, an application can receive from a queue that is local or remote. However, unlike sending, the receiving action is not asynchronous! A queue cannot be opened for receiving unless the machine on which it is hosted is reachable, the queue exists, and the application has the required permissions. Once the queue is open, the application uses the handle returned to make additional API calls which actually receive the messages. Each message is removed from the queue as it is received.

There is one additional wrinkle to this process, involving the use of transactions. Queues can be marked as "transactional" and messages can be sent and received with or without using a transaction. If a queue is marked transactional, then sending messages to it requires the use of a transaction, and a transaction cannot be used if the queue is not transactional. On the receiving side it is a little different: it is always possible to receive without using a transaction, but if one is used, then the queue must be transactional.

These transactions are not end-to-end; that is, they do not flow from the sending application to the receiving application. On the sending side, the messages sent under one transaction will either all be sent or none. Additionally, MSMQ guarantees that messages sent with a transaction will be delivered exactly once (no duplicates) and in order. On the receiving side, messages received within a transaction are not discarded until the transaction is committed. If the transaction is aborted, then the messages are "put back" in the queue and the receiving application can try to receive them again. (Because messages must be in some queue, the messages are not actually removed as they are received, they are just rendered invisible and then either removed or made visible again when the transaction is either committed or aborted.) When receiving, a transaction would usually be aborted because some other operation failed, such as placing the message in a database — you can see where the application would want to keep the messages around and keep trying until the action it was trying to take based on those messages succeeded.

Comments

• Anonymous
  May 23, 2007
  Does the blog have a links section?  It would be cool to know what other queuing blogs/resources are out there.