Issues to Be Aware of When Using COM+ QueuedComponent with Object Pooling

Problem Description

The component is a COM+ event class. It is also a COM+ QueuedComponent (QC) and has object pooling enabled. The clients are a COM+ event publisher. They call into the event component to fire an event. Subscribers are notified of the event. The application works for about a few minutes and then stops working. The event class stops firing any event.

Below is the entry in the application event log while the app stops working:

Event Type:        Error
Event Source:    COM+
Event Category:                (103)
Event ID:              4772
Date:                     2/22/2009
Time:                     2:13:11 AM
User:                     N/A
Computer:          ComputerName
Description:
The COM+ Queued Components Player was unable to create an instance of a Queued Component. CPlayer BindToObject
Server Application ID: {B3DEC044-DFAB-46E0-B424-EBB3FDF90474}
Server Application Instance ID:
{6E75B9BA-DE44-4044-ACE4-CB69F6C73E73}
Server Application Name: ApplicationName
Error Code = 0x8004e024 : COM+ activation failed because the activation could not be completed in the specified amount of time.
COM+ Services Internals Information:
File: d:\srvrtm\com\complus\src\comsvcs\qc\player\player.cpp, Line: 467
Comsvcs.dll file version: ENU 2001.12.4720.1830 shp

Cause

First of all, there is a fundamental issue to be aware of when using QC with a pooled component.

The QC listener in a COM+ server application is designed to scale up to the configured maximum number of threads and it aggressively retrieves messages with as many threads as it has running.  When it first retrieves a message, the message is more or less opaque.  At this point we would not be able to throttle the number of threads based on the maximum pool size of a particular pooled component, for example, or really any contextual information about that component because we don’t know yet what component that message is intended for.  The maximum listener threads is really intended to be the only control on concurrency of QC message processing.

The problem is that object pooling is another control on concurrency in COM+ applications. One of the primary uses is managing limited resources like database connections, and being designed with this in mind, the maximum pool size is really a hard cutoff on the number of instances of a particular component that can exist at any one time.  Although object pooling might also be used to improve performance of some scenarios, this cutoff needs to be kept in mind.  If you configure a component to have a maximum pool size of 12, for example, this is a concurrency limit for that component.

Now let’s look at what happens if the QC listener tries to do more than the object pool can handle. Let’s say QCListenerMaxThreads is 64 (the default for a 4 proc machine = 4*16), and the pool size for one of the components with queuing enabled is 12.  If there are 64 messages in the queue, all intended for the component with max pool size 12, it is possible that 64 threads will be each processing one of these messages. Each of these thread tries to create an instance of the component. Only 12 will immediately get an instance, and the rest will block, possibly until the activation timeout is reached and activation fails with a CO_E_ACTIVATIONFAILED_TIMEOUT (0x8004e024) error. Note that as soon as one of the threads is finished with its instance and returns it to the pool it will almost immediately enable another message to be pulled from the queue, which might also contend for that component. Object pooling uses a semaphore object, which doesn’t guarantee first-in, first-out.

Something along these lines is very likely to be the cause of the event log error. In this particular case, since we failed to even activate the component, the message gets sent to the application’s dead queue.

And it may get worse if there are normal activations coming in for this component as well. These are contending for the same objects as the listener threads, and have no concurrency limit. Whenever you have both normal and queued activations for a component, there is no way to configure it to completely avoid this issue.

Resolution

The component in this particular case is an event class and it shouldn’t need pooling. Compared to components that need to get database connections, etc., pooling probably isn’t buying much in this scenario. Our recommendation is to choose either QC or pooling and disable the other.

The takeaway from all this is that every queued, pooled component in a COM+ application needs to have a maximum pool size at least as large as the maximum number of listener threads to avoid this issue.

We don’t have any KB articles specific to this, but this one gets close to the issue:

PRB: Virtual Deadlock When You Call Pooled Component from Queued Component