Throttling Troubleshooting (Windows Server AppFabric Caching)
When physical memory becomes low on a Windows Server AppFabric cache host, the cache host can enter a state called throttling. The cache cluster will not write data to any cache that resides on a throttled cache host until the available physical memory increases to resolve the throttled state.
Diagnosing Throttling
The most obvious symptom of throttling will come from applications. Attempts to write to the cache will generate DataCacheException errors. For more information about the exception, see RetryLater (Throttled) (Windows Server AppFabric Caching). To confirm that throttling has become an issue on the cache cluster, you can use one or more of the following tests.
Use the
Get-CacheClusterHealth
Windows PowerShell Command. Look to see if any of the cache hosts have caches with percentages in theThrottled
category.Enable the Operational log in the Event Viewer on each cache host, and look for Event 116 which states "Service in throttled state".
Use the Performance Monitor to track the Memory | Available MBytes on each cache host. When this gets under 15% of total physical memory, the cache host enters the throttled state. Throttling also occurs when the cache host memory comes within 4% of the
CacheSize
setting.
Note that a server in the throttled state will also show increased eviction runs. For more information on eviction problems, see Eviction Troubleshooting (Windows Server AppFabric Caching).
Note
For more information about the tools described above, see Health Monitoring Tools (Windows Server AppFabric Caching).
Resolving Throttling
Before deciding on a resolution to throttling, it is important to better understand why it occurred. The table below provides a possible cause and the recommended solution.
Throttling Cause | Description and Resolution |
---|---|
Other processes using too much memory. |
It is possible that other processes on the cache host are using large amounts of memory. This could result in low-memory conditions depending on the amount of memory reserved for the Caching Service. You can detect this by using the Process | Private Bytes counter in Performance Monitor to view the memory used for each process. If the Caching Service, DistributedCacheService.exe, is not the primary memory consumer, look for other processes that are consuming a high percentage of memory. One solution is to move the problem processes to other servers that are not cache hosts. You can also add more physical memory to the machine. Note Although you can use |
Caching Service using too much memory. |
This can be identified by using Task Manager or Performance Monitor to view the memory used by the Caching Service, DistributedCacheService.exe. It is possible that one or more of the caches have disabled expiration and/or eviction. This can contribute to the low-memory condition. You find out the caches in the cluster by using the Windows PowerShell command |
Uncollected .NET memory. |
Although .NET garbage collection occurs automatically, there may be times between collections where uncollected memory is contributing to the low-memory conditions of the cache host. In this scenario, use the Windows PowerShell command |
Custom regions. |
Application can create custom regions with the CreateRegion method. These regions always exist as a unit on a specific cache host. If the application puts large amounts of data in a single region, it is possible that one cache host will become throttled even though there is available memory on other cache hosts. One solution is to add more memory to each cache host in the cluster, because you cannot know for sure which cache host will be chosen for the custom region. The other solution is to redesign the application to store less data in the region or to create and use multiple regions. |
Tag hash tables |
Several AppFabric caching methods allow you to store items in the cache with associated tags. Using tags creates internal hash tables that are not later removed even after the associated items are removed. This is not a memory leak, but it does contribute to the overall memory usage by the AppFabric Caching Service. If an application uses tags that change over time, this could contribute to memory pressure. |
See Also
Concepts
Troubleshooting Server Issues (Windows Server AppFabric Caching)