Common queries regarding Dynamics CRM Asynchronous Service
Some common questions regarding the Dynamics CRM Asynchronous Service
I have two servers which hosts Asynchronous service Server A and Server B which is on a Load balancer. When I look at the Asynchronous jobs on both servers, it never appears as same. At some instances I see one server has 15 jobs and other sitting idle, sometimes vice versa. At the end of the day when I look at the jobs it is not similar.
So what is the reason for the difference in jobs on both Asynchronous servers?
CRM Asynchronous service process the jobs independently i.e. if you have multiple Async servers it will not sync with each other to really balance the jobs.
When each server performed a query they found some work to do. A server could starve for jobs if the other server is able to process and keep selecting jobs with none left for the other server. So you cannot really load balance the Asynchronous service. All you can really do is have overflow. The servers are really fighting each other for work rather than sharing the work.
To better understand I will explain the behaviour giving two scenarios.
Before Jumping on to the scenarios it would better to explain the terms used in that.
AsyncItemsInMemoryHigh – Max number of async operations the service will store in memory. Upon selection interval, if the number of items in memory falls below AsyncItemsInMemoryLow, the service will pick enough to reach up to AsyncItemsInMemoryHigh again.
AsyncItemsInMemoryLow – Minimum number of async operation the service needs to have in memory before loading new jobs into memory. Upon selection interval, if the number of items in memory falls below this value, the service will pick up enough to reach AsyncItemsInMemoryHigh again.
Scenario 1
• AsyncItemsInMemoryHigh is 2000 (This is the default value)
• AsyncItemsInMemoryLow is 1000 (This is the default value)
• AsyncSelectInterval is 5 (seconds) (This is the default value)
• Current jobs ready to be processed in CRM is 5000
• Two Asynchronous Services (AS1(Server A), AS2(Server B))
• Services both start at same time
AS1 and AS2 will start selecting work 5 seconds after service initialization is complete in chunks of AsyncItemsInMemoryHigh – Outstanding Operations. Outstanding Operations are operations currently loaded into memory from previous selects.
Only 1 service should be able to select at a time to avoid both services selecting the same work.
Since both have empty outstanding operation queues they will select the max of 2000 rows each.
Let’s say all the work AS2 selected is long running work and so after 5 seconds it still has 2000 outstanding operations. It will skip its selection of work.
Let’s say all the work AS1 selected is short running work and so after 5 seconds it has processed all of its work. It will select another AsyncItemsInMemoryHigh – Outstanding Operations, in other words 2000 items. There is only 1000 items in the queue so it selects 1000.
As you can see from this example AS1 did 60% of the work while AS2 did 40%. This ratio can continue to slide one way or the other further if one service continues to be unlucky and select work that takes longer, or that when it gets to the table there is nothing to “eat”.
Scenario 2
• AsyncItemsInMemoryHigh is 2000
• AsyncItemsInMemoryLow is 1000
• AsyncSelectInterval is 10 (seconds) (Modified in DB)
• Current jobs ready to be processed in CRM is 5, with 5 new work items introduced every 20 seconds
• Two Async Services (AS1, AS2)
• Services both start at same time.
After 10 seconds AS1 will select all 5 jobs
AS2 will have to wait for AS1 to select its work because AS1 got to the work just a little bit faster, and so AS2 will have no work to complete, and wait another 10 seconds
After 10 seconds 5 new jobs show up.
After 10 seconds AS1 will select all 5 new jobs
Again AS2 will have no work to complete, and wait another 10 seconds
In this case AS1 will do 100% of the work.
How does the CRM Asynchronous service process the job? Is it in batch or one by one?
In your CRM deployment there are two Asynchronous servers and each server will poll the ‘AsyncOperationBase’ table looking for the work to perform. It will select work in batches up to the limit (‘AsyncItemsInMemoryHigh’ this value is stored in the DB) It wants it’s in memory queue to be as full as possible. This can result in 1 server performing more work than the other.
From where does the CRM Asynchronous service pull the jobs? Does it query the Database or the CRM Web service?
CRM Asynchronous service polls CRM Web service every 5 seconds. This is by default and stores this value as ‘AsyncSelectInterval’ in the database.
Let’s move to some CRM tweaks to improve the performance.
Being said that 2 Async servers polls the web service for jobs, we may come in to a situation where two servers actively looking for jobs to process and will certainly at same point will try to process the same jobs causing a deadlock.
You can configure the following registry key on the Asynchronous servers in order to improve performance and avoid dead locks.
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSCRM
Name: AsyncDBAppLock
Type: DWORD
Value: 1
For more information on the above registry key refer to the following KB:
https://support.microsoft.com/kb/2249156
Refer to below article which discuss about CRM 4.0 SQL Dead locks.
If Microsoft CRM Async Service show signs of high CPU usage and high memory usage when a high transaction load and a large number of workflows are in use with Microsoft CRM. You can follow below KB article which discuss this in detail,
https://support.microsoft.com/kb/2489162
Best Regards
Dynamics CRM Support Team
Comments
Anonymous
April 15, 2014
Hi, I saw your posts on CRM. Are you able to explain why CRM benefits from integration with Yammer? I thought CRM already had walls (social functionality)? Is the only benefit that Yammer works for those without CRM licenses? thank youAnonymous
November 23, 2014
Hello, all KB refer to CRM 4.0. Does it also concern CRM 2013 ? Charles