MSMQ and the Windows Firewall

I was called on to assist recently in diagnosing why an MSMQ client was unable to send messages to a queue on a server. One of the things the owner of the machines did was take a network trace on the client of the connection attempt, and what I saw there sent me down the wrong path for a while due to the way MSMQ interacts with the Windows Firewall. To save everyone else the trouble, I thought I'd discuss how the MSMQ installation sets up the firewall to work with MSMQ and why I went astray.

The application in question was using MSMQ's native protocol, so the client was attempting to make a TCP connection to port 1801 on the server. When the client sends the first packet to start the connection process, one of three things will usually happen:

• The server receives the packet, and a process (MSMQ's Queue Manager) is listening on port 1801, so the server sends a reply which continues the connection process.
• The server receives the packet but no process is listening on port 1801, so the server sends a reply which aborts the connection process.
• The server doesn't receive the packet, so there is no reply. The client tries a couple more times, then the connection process times out.

When I looked at the network trace, I could see the client making multiple attempts to start a connection to port 1801 on the server, and the server was not sending back any replies to those attempts, indicating that the network stack on the server wasn't receiving the packets at all. The client had no problem making other kinds of connections to the server, so we knew that the server's networking in general was OK. Selective deafness like that is usually the result of firewall problems, so we briefly went on a wild goose chase looking at the MSMQ-related firewall rules, only to find that they were present and enabled, just as they should be.

Here's the critical bit: when we think of firewalls, we normally think of opening ports, but that is not how the MSMQ installer sets up the firewall rules for MSMQ. Instead, it "whitelists" the mqsvc.exe process for certain network protocols, meaning that (for example) any TCP port which mqsvc.exe opens to listen on causes the firewall to automatically allow packets to that port. This approach was taken because the ports MSMQ uses aren't 100% fixed. This is particularly true of the ports it uses for RPC: MSMQ will automatically use alternate ports if its first-choice defaults are already in use. Other ports can be changed by the use of registry keys. By using these whitelist firewall rules, the firewall automatically adapts to whatever ports MSMQ uses, minimizing surprises and complexity.

The breakthrough in our diagnosing came when we discovered that MSMQ wasn't listening on port 1801, or any of its other usual ports, which quickly led us to discover that it had been inadvertently configured in hardened mode, in which it will only accept messages via HTTP. Since it wasn't listening on port 1801, the firewall wasn't allowing packets to that port, giving the appearance of a firewall problem even though it was doing exactly what we'd asked it to do.

The moral of the story is that, because of the way the MSMQ firewall rules are set up, what appears to be a firewall problem can be a symptom of issues elsewhere. Before you spend too much time staring at the firewall, check that MSMQ is listening on the expected address and port combinations.