"Why are my Express MSMQ messages being written to disk?"

Occasionally someone comes along and gives the foundations of your understanding a good, hard shake. You then have to decide between "shall I just pretend I wasn't listening?" or "do I feel an evangelical zeal to let everybody know the news?"

For some reason I went for the latter.

So, the bedrock of MSMQ is messages and how it stores them. I've been telling customers for years that Express messages were kept in memory and only Recoverable messages were additionally persisted to disk. The KnowledgeBase documentation at the time (it should be being updated sometime soon)  seemed to back this up and nobody complained.

Unfortunately this isn't quite the case. Express messages are written to disk just like recoverable messages with only one main difference:

	Express messages	Recoverable messages
MQSendMessage API (or "Send" in Com / System.Messaging)	Returns immediately to the caller a s messages are written with "lazy writes" so it is not guaranteed that they are on the disk if there is a crash soon after.	Returns only after it receives a storage acknowledge which is created when the message is physically saved to the disk (MSMQ Queue Manager calls FlushViewOfFile) and not in any disk cache.

This is why recoverable messages are significantly slower than express messages in single thread scenarios but only marginally slower in a multiple threads scenarios. 

All MSMQ messages are written to memory mapped files in the Storage directory:

• R*.MQ files are for Express messages
• P*.MQ files are for Persistent (Recoverable) messages, including Transactional messages
• J*.MQ files are for Journal messages
• L*.MQ files are bitmap indexes to the Persistent and Journal files

What is the bitmap index for? MSMQ uses this file to reload messages into memory on startup. Each 4Mb storage file is a bit like Swiss cheese - over time holes start appearing where messages have been read but the space left behind hasn't yet been filled by new messages. The bitmap index simply flags blocks in the file as in use or empty. On startup, the storage file is read "as is" into memory along with the index so that MSMQ can remember where it put the messages last time and where there is space to use.

Note that Express messages don't have bitmap indices because the R*.MQ files are not supposed to be reloaded into memory on startup. As Express messages are written to disk without waiting for confirmation that the write was successful, it is not possible to trust the data enough to use a bitmap index to reload messages back into memory on startup. Which also explains why MSMQ deletes all the R*.MQ files it finds in the Storage directory on startup or shutdown. 

References

174307 Interpreting file names in the Storage directory in MSMQ