Understanding Client Throttling

Microsoft Exchange Server 2007 will reach end of support on April 11, 2017. To stay supported, you will need to upgrade. For more information, see Resources to help you upgrade your Office 2007 servers and clients.

 

Applies to: Exchange Server 2007, Exchange Server 2007 SP1, Exchange Server 2007 SP2, Exchange Server 2007 SP3

With the release of Microsoft Exchange Server 2007, a new feature known as RPC Client Throttling is available to administrators to help manage the end-user performance experience. RPC Client Throttling was introduced to help prevent client applications from sending too many RPC operations per second to the Exchange server which could decrease overall server performance. These client applications include desktop search engines searching through every object inside a user's mailbox, custom applications written to manipulate data that is located in Exchange mailboxes, enterprise-class e-mail archiving products, or CRM enabled mailboxes with e-mail auto-tagging enabled. Client throttling allows Exchange to identify and help prevent server monopolization by a few users. When a client is identified by the Exchange server as causing a disproportionate effect on the server, the server will send a "back-off" request to the client to reduce the performance effect on the server.

Note

In Microsoft Exchange Server 2003, there was a registry key named “Disable MAPI Client Processes” as described in A feature is available to disable MAPI program access to a computer that is running Exchange Server 2003. This registry key could be enabled to help lock down certain processes from connecting to the Information Store service. This registry key is no longer implemented in Exchange 2007 and has been replaced with this new RPC Client Throttling mechanism.

How RPC Client Throttling Works

If RPC average latency on the Exchange 2007 server is higher than normal, Exchange server users may start to experience noticeable service degradation. RPC average latency should be kept below 100 milliseconds in order to maintain an acceptable end-user experience.

RPC Client Throttling uses a computational method that is based on overall RPC average latencies and remote operation (ROP) statistics to determine whether a given client will start receiving back-off requests to limit the number of operations that the client can perform against the server. This method computes the average delay between ROPs over three sampling periods for a given minute. This logic uses a rolling average of the ROP rate for the last minute. Once this value is determined, it is then compared to the overall RPC average latency. If this computed value is lower than the overall RPC average latency, then back-off requests will be sent to the client.

Important

Short spikes of high ROPS per second will not result in a back-off request being sent to the client as long as the average RPC latency for the last minute is low.

For example, if you have an RPC latency of 25 milliseconds, a well-behaved client can successfully send up to 40 ROPS per second (40*25=1000 milliseconds) with no throttling. If RPC latency increased to 100 milliseconds, and the client attempted to send the same amount of ROPS, the client would then start receiving back-off requests from the server. From this point, the server will only allow the client to send up to 10 ROPS per second until the next RPC average latency sample is again below 100 milliseconds.

The logic for back-off requests has two main parts. The first is how the server decides to throttle the client and the second is the determination to block the client from accessing the server. Inside the Information Store service, there is a client activity log that keeps track of each user’s activity against any given server. Each client activity log is associated with a users MAPI session. If the server makes a decision to throttle a client, a back-off request is added to the session back-off queue for that user's MAPI session. This back-off information is then used to determine if the client request is accepted or rejected by the server. If the user has multiple mailboxes open for a single MAPI session, this back-off information will apply to all mailboxes for that MAPI session.

Depending on the version of Microsoft Office Outlook being used, the server will send different back-off information to the client's RPC response stream.

• For Microsoft Office Outlook 2007 clients, a ropBackoff request is added to the user's back-off queue. This request contains information regarding the current back-off delay, and tells the client to delay sending any additional requests to the server for a specified time. The client will then retry sending requests after the specified time expires. The maximum time for this back-off delay is hard coded to 2000 milliseconds or two seconds.

  Note

  We recommend using Outlook 2007 clients against an Exchange 2007 server for an optimal client experience.

  Note

  The ropBackoff is a new feature in Outlook 2007 and the earlier Outlook clients do not understand this function.

• For Microsoft Office Outlook 2003 or earlier versions, a status code of RPC_S_SERVER_TOO_BUSY is sent. This status code mimics the response Exchange 2003 servers send to clients when Information Store RPC threads have been exhausted. The earlier versions are designed to catch this exception and then wait for a length of time before the call is retried as specified in the response from the server to the client. The default time for this delay is one second. If the client tries the request every second for one minute and still receives the same exception each time, the client gives up and disconnects its session to the Exchange server.

Monitoring for High RPC Operations per Second Conditions

To help detect when clients are being backed off, the performance counter “MSExchangeIS\RPC Client Backoff/sec” can be monitored on Exchange servers that have the Mailbox server role installed to determine the rate at which back-off requests are occurring.

Counter	Outlook 2007 expected values	Outlook 2003 and earlier expected values
MSExchangeIS\RPC Client Backoff/sec The rate that the server notifies the client to back-off.	50 per client	1 per client

For Outlook 2007 clients that are being throttled, you can expect the “MSExchangeIS\RPC Client Backoff/sec” counter to show approximately 50 back-offs per second for each client. For Outlook 2003 and earlier versions, you should expect, at most, one back-off per second for each client. The difference in the expected rates is caused by the granularity of the back-off method that is used for the respective clients. The smallest back-off period for Outlook 2003 is one second, whereas the smallest back-off period for Outlook 2007 is one millisecond.

When back-off information is sent to the client, the data contains two parts. The first part is the time that the back-off was inserted. The second part is the duration of the back-off period. The duration of the back-off period is calculated based on the following formula: (MSExchangeIS\RPC Average Latency * RPC throttling factor)/1000. By default, the RPC throttling factor is a value of 1000. This means that the back-off period is exactly the value of RPC average latency. The back-off value that the server sends to the client is the number of milliseconds until the back-off period expires. If the client sends another operation to the server before the specified back-off period expires, the server will return another ropBackoff with an updated delay value. This will continue to occur until the back-off period expires, thus removing the ropBackoff from the user's back-off queue. A sample of what this resembles in Performance Monitor can be seen in the following graphic:

Tuning for High RPC Operations per Second Conditions

On Exchange servers that have the Mailbox server role installed, a registry key is available to help tune the throttling computation. This registry key is called RPC Throttling Factor and is located under the HKLM\System\CurrentControlSet\Services\MSExchangeIS\ParametersSystem key.

Note

This value is not set by default and must be added as a DWORD value under the previously mentioned key.

[HKLM\System\CurrentControlSet\Services\MSExchangeIS\ParametersSystem]

RPC Throttling Factor

DWORD: Can range between a value of 0-5000

To show how modification of the RPC Throttling Factor works, consider the scenario where you want to configure the server to backoff clients for a longer period of time, such as 2 x RPC Average Latency. You can do this by setting the RPC Throttling Factor to a value of 2000. This will increase the backoff period for each MAPI session by a factor of two. In this case, if RPC Averaged Latency was 50 milliseconds, a throttling factor of 2000 (50 ms * 2000/1000) will equal 100 milliseconds. Each backoff request will be extended to 100 milliseconds instead of the default 50 milliseconds, thus delaying additional client MAPI requests to the server for 100 milliseconds. Understand that such a configuration change may appear to the client as slow server performance. However, if this client has an adverse effect on overall server performance, resource usage on the Exchange server by this client will be minimized. This improves perceived server performance for all other users on the server.

Important

If this problem should occur for many or all clients on the server, then a server-side investigation should be performed to detect any bottlenecks that may be occurring.

For an optimal experience with this throttling mechanism, a minimum of Outlook 2007 and Exchange 2007 is recommended.

You can disable throttling altogether by setting the throttling factor to a value of 0.

Important

We do not recommend disabling client throttling in the Information Store. Making this change will allow poorly behaving clients to degrade performance on any given server.

If you believe that you may have to disable client throttling, we recommend that you first determine the cause of the performance bottleneck on the server. In most cases, identification and resolution of the cause will resolve the need for the server to send back-off requests to the clients.

For More Information

For information about how to troubleshoot slow RPC request processing issues, see Troubleshooting Slow RPC Request Processing Issues.

For information about how to enable client-side monitoring, see How to Enable Client-Side Monitoring.