All System jobs are 'waiting' in CRM 4.0

Why all the System jobs are in waiting status in CRM 4.0?

I have always noticed many of my customers complaining about a very common scenario in CRM where the system jobs are always in “waiting” status. Though there can be many reasons for this behavior I thought of writing about the most common scenario I have ever noticed. It is pretty easy to approach this problem if we know how system jobs are processed in CRM. In CRM 4.0 we have got a pretty neat windows service called “Microsoft CRM Asynchronous Processing Service”. One of the many functions of this service is to process the system jobs. So always ensure that this service is started.

How Microsoft CRM Asynchronous Processing Service process the jobs?

The service will build its own URL to hit the CRM web service to process the jobs. In any case if it cannot hit the web service properly after building the URL the job will go to waiting.

Why all the system jobs are in waiting status then?

The reason is simple as asynchronous service is building a wrong CRM web service URL.

How does the service determine the URL?

There are 3 key values which determine the CRM web service URL built by Asynchronous service:

ADRootDomainScheme:

This will tell the asynchronous service about the protocol (HTTP or HTTPS) to be used while hitting the web service. This value is mentioned in the “DeploymentProperties” table in the MSCRM_CONFIG database under column “NVarCharColumn” corresponding to ColumnName “ADRootDomainScheme”

AsyncSdkRootDomain:

This will tell the asynchronous service the hostname / IP address to be used while hitting the web service. This value is mentioned in the “DeploymentProperties” table in MSCRM_CONFIG database under column “NVarCharColumn” corresponding to ColumnName “AsyncSdkRootDomain”. On a default CRM installation this value will be empty and Asynchronous service interprets an empty value as loopback address - 127.0.0.1

LocalSdkPort:

This determines the port to be used while hitting the web service. This value is mentioned as a registry key under “HKLM\Software\Microsoft\MSCRM” in the server where asynchronous service is running (This will be the CRM application server unless you have a split server role installation where you might have a dedicated server running asynchronous service). In a typical CRM installation if you select “Default website” in IIS for installing CRM, then this value will be 80 and if you select a new website the value will be 5555 unless you change it manually.

Some examples on how asynchronous service builds the web service URL:

Scenario1:

ADRootDomainScheme – HTTP
AsyncSdkRootDomain – empty
LocalSdkPort – 5555
URL built: https://127.0.0.1:5555/MSCrmServices/2007/CrmService.asmx

Scenario2:

ADRootDomainScheme – HTTP
AsyncSdkRootDomain – empty
LocalSdkPort – 80
URL built: https://127.0.0.1/MSCrmServices/2007/CrmService.asmx

Scenario3:

ADRootDomainScheme – HTTPS
AsyncSdkRootDomain – empty
LocalSdkPort – 443
URL built: https://127.0.0.1/MSCrmServices/2007/CrmService.asmx

In Scenario3 you might have configured the CRM website to accept requests only on HTTPS by installing an SSL certificate and configuring IIS to accept only HTTPS requests. In that case you need to change the ADRootDomainScheme and LocalSdkPort values accordingly so that Asynchronous service can hit the correct web service URL. Please note that the asynchronous service can parse the SSL certificate warnings so it is not required to populate AsyncSdkRootDomain with the SSL cert name. But it is recommended.

Scenario4:

ADRootDomainScheme – HTTP
AsyncSdkRootDomain – IPaddress (of CRMServer)
LocalSdkPort – 5555
URL built: https://IPaddress:5555/MSCrmServices/2007/CrmService.asmx

In Scenario4 you might have configured the CRM website under a dedicated IP address. You need to change ADRootDomainScheme and LocalSdkPort accordingly if the same website is locked with an SSL certificate.

Scenario5:

ADRootDomainScheme – HTTP
AsyncSdkRootDomain – host header name
LocalSdkPort – 5555
URL built: https://hostheader:5555/MSCrmServices/2007/CrmService.asmx

In Scenario5 you might have configured the CRM website to work with an alias host name by adding a host header to CRM website and removing the default unassigned binding in IIS. You need to change ADRootDomainScheme and LocalSdkPort accordingly if the same website is locked with an SSL certificate.

Scenario6:

In a load balanced environment it is recommended to populate AsyncSdkRootDomain with the load balanced URL / alias name

Some tips on troubleshooting:

I always prefer to check if all the system jobs including workflow jobs are in waiting or if it happens only to specific system jobs. If it happens to only specific system jobs then the reason can be different as those jobs in succeeded status clearly tell us that the asynchronous service is building and hitting the web service URL in a correct fashion. It is a good to confirm this by creating a sample workflow like one which will create an activity on creation of an account / contact. Another placeholder to check is the IIS logs for the CRM website. If we check the IIS logs we are likely to see entries as:

#Software: Microsoft Internet Information Services 6.0

#Version: 1.0

#Date: 2010-01-27 00:02:44

#Fields: date time s-sitename s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status

2010-01-27 00:02:44 W3SVC2 127.0.0.1 POST /MSCrmServices/2007/CrmService.asmx - 5555 - 127.0.0.1 Mozilla/4.0+(compatible;+MSIE+6.0;+MS+Web+Services+Client+Protocol+2.0.50727.1433) 401 2 2148074254

2010-01-27 00:02:44 W3SVC2 127.0.0.1 POST /MSCrmServices/2007/CrmService.asmx - 5555 - 127.0.0.1 Mozilla/4.0+(compatible;+MSIE+6.0;+MS+Web+Services+Client+Protocol+2.0.50727.1433) 401 1 0

2010-01-27 00:02:44 W3SVC2 127.0.0.1 POST /MSCrmServices/2007/CrmService.asmx - 5555 REPLICA\CRMREPLICA$ 127.0.0.1 Mozilla/4.0+ (compatible; +MSIE+6.0;+MS+Web+Services+Client+Protocol+2.0.50727.1433) 200 0 0

In the above case the AsyncSdkRootDomain value is set as empty (default) hence the source IP is loopback and CRM website is on port 5555 with an “all unassigned” binding in IIS. Note the sc-status / sc-substatus codes. 401.2 is expected as CRM website is not configured for anonymous authentication followed by 401.1 which is an NTLM challenge followed by 200 (Status OK/authenticated) code shows us that the request made to CRM web service is a success.

I also prefer to do an advance find on those system jobs (https://blogs.msdn.com/benlec/archive/2009/02/10/how-to-troubleshoot-system-jobs-failures-using-advanced-find.aspx) on waiting status after customizing my advance find view to add columns “Error Code” & “Message”.

As always my last option would be a CRM platform trace.

Cheers,
Nithin