Share via

Enterprise Library 5 Logging using MSMQ

  • Article

In designing a new application, I encountered the issue of building a reliable and high performance distributed logging.  After much research, the decision was to use Enterprise Library 5.0 with MSMQ distributor service.

 

Since EL 5 does not support asynchronous logging, the closest thing to asynchronous logging in EL 5 is MSMQ listener.  In a nutshell, MSMQ trace listener is to log messages into MSMQ.  Taking these messages out of MSMQ and put them into a different storage (file, event log, database) is the job of the MSMQ Distributor which is part of the EL 5.

 

I searched internet and did not find any articles with detailed and step to step guide to implement this.  So here it goes.  I assume you have some basic understanding of EL5 logging.

 

First of all, our requirement is to log application running across multiple machines to a single SQL 2008 R2 database.  Each machine's load changes a lot so it is necessary that the logging will be the bottleneck in peak hours.  Log storage must be centralized, multiple concurrent writes should not cause lock or resource contention.  When centralized database is down, the log should continue to accumulate and write to the database when it is resumed.  Finally, It is ok to occasionally log twice, but log should not disappear.

 

Base on the requirements above, we chose EL 5 MSMQ Distributor Service.  All features I describe here is all out of box. 

 

Architecture Layout

 

Our design is illustrated in the following diagram (taken from https://msdn.microsoft.com/en-us/library/ff647847.aspx).  Computer A, B and C use MSMQ trace listener to write log to a private queue on computer C.  MSMQ Distributor Service is Windows service.  Once configured and started, it takes log entries from the queue and writes to central log database (not in the picture)

 

 

The main flow is like this:

 

1. Application on machine A calls EL logging API to write a log entry

2. EL serilizes the entry using binary formmater and drops the serilized log entry into local MSMQ outgoing queue

3. MSMQ sends the entry as a message to destination queue on computer C

4. MSMQ distributor service polls the destination queue

5. MSMQ distributor service reads the message and deserializes it back to log entry

6. MSMQ distributor service calls EL API to log the entry

7. MSMQ distributor service removes entry from queue

8. All computers are in the same domain

 

 

Installation

 

Install MSMQ (both client and server).  On Windows Server 2008R2, MSMQ is a service under Application Server role.

1. Message Queuing Server

2. Directory Service Integration - this is necceary for message level security signing or encryption where a certificate is needed

 

 

 Once MSMQ installed, register user certificate.  This is only necessary if you need message level security. 

One MSMQ is installed, create a non transactional private queue where all logging messages will be sent.  Transactional queue will work but not necessary since MSMQ Distributor does not use distributed transaction.

 

After queue is created, set permissions as needed as who has what access to the queue.

Configuration

There are two .NET configuration files for logging.  One is on the client side where applications write log to MSMQ queue.  The other configuration file is on the serve side for MSMQ Distributor.  Use EL5 configuration editor if possible.

 A few things worth mentioning:

1. queuePath - this is to instruct EL logging to use it as a destination queue.  If the queue is not running, message will be queued locally in outgoing queue.  Pay special attention to syntax

2. formatter - must use binary formatter!  MSMQ Distributor uses binary formatter to read and deserilize message in queue

3. recoverable - this is to instruct if message should be persisted on disk.  If this is set to false, then messages in queue will not survive shutdown

4. timeToReachQueue - how long message should remain in outgoing queue if destination queue is unreachable.  After that time elapsed, the message will either be in system deadletter queue if useDeadLetterQueue is set to true, or discarded.  The message will be in deadletter queue on the source computer   Default is about 250 years. 

5. timeToBeReceived - how long message should remain in the destination queue.  The message can remain there for long time if MSMQ Distributor service is not running.  The message will be in deadletter queue on the destination computer 

 

6. useAuthentication - this is to enable message authentication and should match queue's authentication setting.  Also both sender and receiver need to have updated certificates

7. Make sure you have error source configured.  In above example, error source uses event log trace listener.  Error source is used as an alternative when error occurs.  For example, if MSMQ service is not running, writing to queue will fail.  However, this error will not cause EL5 logging API to throw exception (for apparent reason).  If there is no error source, then message is lost.  In above example, all failed MSMQ logging will be written to event log

8. If you have multiple listeners for same category source, order of listeners is important.  Recommendation is to put more reliable ones in front.  Any listener exception will cause EL 5 to skip rest listeners in pipeline. 

MSMQ Distributor Configuration

MSMQ Distributor is a Windows service that takes messages from destination queue, and use EL 5 logging API to write log.

Nothing special here other then all the log is written to database called Instrumentation.  Make sure the MSMQ Distributor service has permission to write to it.  The SQL script to create database is part of EL 5 source (EntLib50Src\Blocks\Logging\Src\DatabaseTraceListener\Scripts).

Once this is configured, you can call Logger.Write(logEntry) to log messages in database.

Comments

  • Anonymous
    August 31, 2011
    Mark - Thanks for the post I have the distributor service running, but my app with enterprise library logging is not writing logs to queue, I verified the queue by writing a test message into the queue. On further debugging I found that the app writes Text Formatted messages but not binary messages, any ideas ?

  • Anonymous
    December 05, 2012
    A suggestion to avoid a single point of failure and potential latency introduced to the application due to network traffic, you should consider having each computer write to a local MSMQ rather than all computers writing to a single computer's MSMQ.

  • Anonymous
    January 13, 2013
    Hi Mark, Thanks for the post, i refered your post to implement Asyn loggin and it is working fine as i have followed all the step described in your post. Now I have to implement Custom logging for that I am using "ExtendedFormattedDatabaseTraceListner" and modified the MsmqDistributor.exe.config file but not able to do that. I am getting config error whil start the Service. Please provide your valuable input in this regards.

  • Anonymous
    August 26, 2014
    Next time paste text instead of a bitmap so we can copy.

  • Anonymous
    July 20, 2015
    Hi Mark, I am unable to install MSMQ Distributor  and the error is can not load the file or its dependency. Do you know what dependency it has? I am using EL6, which does not include the msmqdistributor.exe.