“Slow Performance” accessing CRM IFD published with ISA/TMG
In this article I want to provide a detailed view on a possible performance issue you might face, when you’re publishing CRM 2011 IFD using ISA Server or TMG.
Please note that the screenshots provided in this article are based on TMG, however the configuration in ISA is very similar to the configuration described in this article.
This article won’t cover the details on how to publish the CRM IFD (Internet Facing Deployment) in detail. In this article I assume, that you published your CRM IFD to the Internet using a standard Web Publishing rule on ISA/TMG and that the access to the CRM IFD is working from the Internet.
Scenario:
Consider the scenario, where your CRM IFD is up and running. You setup the publishing rule in TMG and everything appears to work as expected. Now you start sending the information about the published CRM to your users in their home office. A few minutes later you receive the first calls from users complaining that accessing the CRM IFD is terribly slow from their home office.
Troubleshooting:
To verify if something is going wrong with the server, you connect to the CRM IFD site, and cannot see any issues. Next you connect to the CRM Server and to the TMG to verify if you can see any obvious performance issues with the server, but everything appears to be “green”. In order to understand why the user experience is different to your experience you should start tracing the issue from the client which reported the issue.
One good way to perform this tracing is to collect a network trace on the client in order to verify if there’s anything like network delays which might be causing the issue.
I assume that your clients access the published site using HTTPS, therefore the relevant traffic in the network trace will be encrypted. If you don’t see any obvious delays you need to use tools to inspect the HTTPS traffic.
One of many tools which can help in this scenario is Fiddler (Download www.fiddler2.com) Please make sure to configure Fiddler to decrypt the SSL traffic as described here:
http://www.fiddler2.com/Fiddler/help/httpsdecryption.asp
Once you setup fiddler on the client, please ask the user to reproduce the access while tracing the traffic with Fiddler.
Note: Please let the user know, to use a test account to reproduce the issue, as you will be able to see the user password in the fiddler trace.
Once the user collected the data, please ask the customer to save & send the fiddler trace to you. You can open it using Fiddler.
As the Fiddler trace will most likely not only include the traffic to the CRM host, but also some “noise”, you should first sort the tracing by host name and save only the relevant traffic between the client and the CRM server in a separate file.
With all the noise filtered it’s a good start to select all packets and have a look at the statistic section provided by Fiddler (screenshot below)
When you have a look at the ACTUAL Performance in this case, you can see that the client spends around 1:47min to load everything:
Requests started at: 10:23:13.759
Responses completed at: 10:35:04.469
Aggregate Session time: 00:01:47.804
The most interesting part is that the client did load about 10MB!
Request Count: 269
Bytes Sent: 141,293 (headers:127342; body:13951)
Bytes Received: 10,347,456 (headers:96136; body:10251320)
Given the fact that the client most is most likely connected with a slower bandwidth, compared to your company connection this explains why you didn’t face the issue when accessing the site!
The question you have to answer now is:
WHY DOES THE CLIENT HAS TO DOWNLOAD 10MB TO VIEW THE PAGE?
The first thing you should verify is if you can see some big files being transferred or some interesting content types in the request.
When you look at the response bytes, you can see which content-types are the “TOP TALKERS”:
RESPONSE BYTES (by Content-Type)
--------------
text/javascript: 6,260,298
application/x-javascript: 2,545,850
image/png: 854,138
text/x-component: 202,974
text/xml: 153,527
[…]
Thinking about this, this appears to be a quite lot of text/javascript and application/x-javascript which is send to the client. To get a better feeling, if there’s a specific pattern behind this, you should sort the Fiddler trace by Body
By looking at the biggest Reponses you can see the following, when you look at the headers:
The client Accepts compression, but for some reason TMG didn’t return the traffic compressed. When you have a look at the payload, you can see that it’s pure text, perfect to be compressed. You can have a look at some different packets, and you’ll see the same behavior. The client accepts encoding, but TMG just return uncompressed packets in most cases.
Time to look at the TMG now, to answer the question: WHY ISN’T TMG COMPRESSING THE TRAFFIC?
First of all you have to verify if compression is enabled. This can be verified when you open the Web Access Policy and open the HTTP Compression settings:
Next you should select the “Return Compressed Data” tab, to verify if the Listener you use in your CRM web publishing rule is listed there. It should be listed by default, the listener is called CRM Test in this case:
The next thing you should verify is which Content Types will be compressed at all, by clicking the Content Types … button.
This will show the following list of content types:
As you can see TMG doesn’t compress all content types by default, only those which are configured in HTML Documents and Text.
To verify what’s configured in those two content types, you have to open the properties of the content types using the Toolbox:
As you can see neither in HTML Documents nor in Text you can find an extension or content type which matches the “TOP TALKERS” from the Fiddler trace. That’s why TMG isn’t returning the traffic compressed.
You may ask yourself now, why isn’t TMG compressing everything by default to save bandwidth?
Here’s a good answer from TechNet:
In general, some Web servers, when responding to a request, do not accurately provide the content type in the response header. For example, a response may include a Microsoft Office PowerPoint® 2007 (.pptx) file, but the response header may indicate that the content is plain text. When an Internet Explorer client receives this type of response in compressed format, it cannot interpret the response, and the user sees meaningless characters on the monitor. If the response is received uncompressed, Internet Explorer can interpret it, and the user can open and view the content.
http://technet.microsoft.com/en-us/library/cc995227.aspx
Button Line: It’s all about user experience. The scenarios where compression can break a website are more common than the scenarios where it helps to improve the user experience.
Note: By default TMG won’t ask for compression, when you use it in as forward proxy configuration, to provide the best user experience.
Of course there can always be scenarios, where you might need to add specific content types to the compression List, in order to provide a better user experience. Publishing CRM 2011 IFD is one of these scenarios.
In order to enable the compression for the “TOP TALKERS”, please do the following:
1. Configure a new Content Type set, including the content types which are listed in the TOP 3 of the TOP Talkers:
application/x-javascript
text/javascript
text/xml
2. Open the Compression settings -> Return compressed settings -> Content Types… and select the CRM content type you just created.
3. To verify if the changes had been applied correctly, you can start another fiddler trace from your client. When you use the Inspectors again, you can see that the RAW content is compressed / non-human readable.
Access to the CRM should be much faster now for users with slow bandwidth.
I hope you can find this article useful.
Update:
We found that some customers experienced some performance issues, even after applying the changes as described in this blog article.
When investigating in this performance issues, we found that you can add another performance boost to the client access when you enable the compression between ISA/TMG and the CRM server.
By default ISA/TMG won’t ask for compression for the traffic between ISA/TMG and the published web server. This is because compression always causes some CPU overhead (compression / decompression) on the Web Server and on TMG, and compression to save bandwidth usually isn’t needed in current LAN infrastructures.
In order to enable the compression between ISA/TMG and the CRM Server, you have to open the HTTP Compression preferences, and select the “Request compressed data” Tab this time.
Add a new Computer Object, which describes your CRM Server, and add it to the list:
Requesting compressed data, changes the way the CRM server generates the responses. With these changes we could observe a much better caching behavior on the client. Which results in a much smaller number of requests the client has to send to the published CRM IFD Web site.
We’re currently investigating in the details of this behavior. We will provide more detailed information once we’ve completed our internal investigations for this behavior.
Author
Philipp Sand
Microsoft CSS Forefront Security Edge Team
Comments
Anonymous
January 01, 2003
This solution greatly improved performance of our web services, because our web listener wasn't listed at all.Anonymous
January 01, 2003
If you use Microsoft Dynamics CRM 2011 Outlook clients or have client side Microsoft CRM SDK applications, you should enable Dynamic http application SOAP+xml compression per this command, %systemroot%System32inetsrvappcmd.exe set config -section:system.webServer/httpCompression /+"dynamicTypes.[mimeType='application/soap%%u002bxml;charset=utf-8',enabled='True']" /commit:apphost , which should be run when included in a batch file. This information is included in the Whitepaper, "Optimizing and Maintaining the Performance of a CRM 2011 Server Infrastructure", available at www.microsoft.com/.../details.aspx . Note that if you wish to run this directly from a command prompt on the server, this part of the command, 'application/soap%%u002bxml; should be changed to only have a single percent symbol, like this, 'application/soap%u002bxml;. In addition, if you are able to setup https encryption on the Microsoft CRM Web site, you will see additional gains in reduced data that needs to be sent that will outweigh any loss from the https overhead.Anonymous
August 07, 2011
FYI, CRM Update Rollup 3 also addresses some compression issues: "Assume that you enable the http compression setting on the Dynamics CRM server. In this situation, the Microsoft Dynamics CRM 2011 Outlook client does not receive responses for SDK calls as compressed. This problem occurs because the Outlook client does not send an "EnableDecrompression" http header with the request." support.microsoft.com/.../2547347Anonymous
April 04, 2014
We've had a couple of recent cases that have come back to this and have found a great resource thatAnonymous
March 17, 2015
We applied these compression in our environment, and it really made a difference. However, we have multiple listeners setup in the TMG farm. When the new content type was introduced to the farm, it applied to all the listeners. This was not exactly how we wanted as we have no ideas how this will impact other applications. The only way to work around this was bypassing the TMG farm. Do you know of any ways to separate compression types per listener?Anonymous
November 27, 2015
I have the same issue as Mike. This is excellent and works for CRM but I need it on a single TMG publication and not on all. Any hepl available please?