通过

Office 2013 Lag – Slow Launch on a Closed Loop Network

  • 项目

Day 1:
Today my customer is experiencing an issue with multiple lag problems running Office 2013 on a Closed Loop Network.  What is a Closed Loop Network?  Closed Loop Network (which I will reference as CLN from now on) is a network that has NO ability to connect the Internet.  Let’s breakdown the scenario:
1) Running Windows 7 w/SP-1.
2) Running Office 2013 w/SP-1.
3) Running McAfee antivirus.
4) Running Internet Explorer as the default browser.
5) Bitlocker is being used for hard drive encryption
6) All workstations are domain joined.
7) All Domain Controllers are running Server 2012R2 (DFL/FFL is also at 2012R2).
8) Home Drives (H-Drives) are mapped to NetApp appliances.
9) Folder Redirection for My Documents also points to the same NetApp appliance.
10) No roaming profiles.
Here is what ALL customers are experiencing:
1) Customer Logs in.
2) Customer launches Word 2013 (I just picked this application as an example).
3) At the Word 2013 “splash screen”, you see one of the following plug-ins load and sit there for up to 30 seconds (per plug-in):
a. OneNote 2013.
b. Adobe Acrobat XI.
c. Dragon Naturally Speaking.
4) Once all the plug-ins finish, then the customer can utilize Word 2013.
a. If they were to close Word and re-open Word, the behavior happens all over again.
5) This is happening to all main Office 2013 applications (Outlook, Word, Excel & PowerPoint, but not Access).
6) Obviously not everyone has all 3 of the above “problem” plug-ins installed but for those that do, they are experiencing up to a 90 second lag every time they launch an Office application.

Now I just love spoilers, so I am going to tell you what the culprit/root cause is:
CRL Checking.
The how/what/why/who/when/where I will get in to on Day 2, which I will post tomorrow.
………… to be continued.

 

Day 2 (sorry for the delay):

Let me start off by saying that 2 of my most commonly used tools are ProcMon (part of the SysInternals Suite) and NetMon (but this has been replaced by Microsoft Message Analyzer).  ProcMon (Process Monitor) is great at giving you a SUPER detailed report of every process that is running on your computer and what it is try do (Query/Write/Open a file or registry key).  NetMon (Network Monitor) is great at giving you a SUPER detailed report of every process that is running that is accessing a Port and/or Protocol.  The reports that these utilities create can be overwhelming.  I ran both of these tools (not going in to detail about the use of these great utilities), one at a time, and started digging through the results.  These are the "key" items that I observed:

  1.  When I launched Word (with Dragon Naturally Speaking installed) I saw DNS traffic looking for crl.verisgn.com.
  2. When I launched Word (with OneNote 2013 previous launched, this is KEY) I saw DNS traffic looking for crl.microsoft.com (NOTE:  I later found out that this also happened with Visual Studio installed)
  3. When I launched Word (with Adobe Acrobat XI installed) I saw DNS traffic looking for crl.symcb.com & crl.symdb.com & ocsp.symantec.com.
    1. NOTE:  symcb and symdb are simply variants of Symantec.

Well on a CLN (Closed Loop Network), this type of traffic is NEVER going to resolve and the websites will never be reached and therefore the servers at the other end of these request will never respond.  The most important part of these URLs are the CRL (Certificate Revocation List) and OCSP (Online Certificate Status Protocol) section.  Without going in to SUPER detail, these products are trying to contact a server to find out if the certificates that were used to sign their perspective company's files are valid or not.  Since there is no server to talk to, they wait XXX number of seconds, then timeout.  XXX number of seconds = LAG!  Each plugin that is installed is that much more LAG experienced.  If each plug-in take 30 seconds and you have all 3 products installed, it could take Word over 90 seconds to launch before the customer can type his/her first letter.  Well this is highly unacceptable.

So now that we know that the Certificate checking is causing the lag there are 3 more things we need to figure out.

  1.  What files are causing this Certificate checking?
    1. Well I left my list of DLL/MSI/EXE files for each product at work, so I can't provide that this today but I will have then added to this BLOG by the end of the week.
  2. Are there any other websites/URLs that these plug-ins are trying to reach that my captures may have missed?
    1. HECK YES!
    2. Between the 3 plug-ins there were almost 20 different URLs that these products were trying to access (that I captured).
  3. How do eliminate/reduce the LAG time of the Certificate checking that will NEVER work?
    1. This is the most important question on this entire web page.
    2. There are over 8 different ways to attack this, but I am only going to focus on the 2 that I used

Option #1 (AKA: The down and dirty, I washed my hands for hours after doing this and never felt clean):

NOTE:  I highly recommend to NEVER DO THIS long term and only use this method as a Proof Of Concept/testing.  Also NEVER do this on an Internet connected computer!

  1.  HOSTS. file
    1. Using the HOSTS. file (no extension), located at the C:\Windows\System32\Drivers\ETC folder, I added the following "DNS" entries:
      1. 127.0.0.1     crl.microsoft.com
      2. 127.0.0.1     crl.verisign.com
      3. 127.0.0.1     crl.symcb.com
      4. 127.0.0.1     crl.symdb.com
      5. 127.0.0.1     ocsp.symantec.com
    2. I then saved the HOSTS. file (by default it is read only to a "normal" user).
    3. I re-launched Word and the launch time, with all 3 plug-ins installed, was between 5-10 seconds (before it was over a minute long).
      1. I tested this with Excel, PowerPoint and Outlook and they all had the same speed improvement.

So what did I do/accomplish by modifying my HOSTS. file?  I told the software that when you go looking for these EXACT (notice there are no wildcards being used here) websites that you will "call home".  127.0.0.1 is every computer home address, no matter what its true IP address is.  So the Certificate checking was looking for crl.microsoft.com, and the HOSTS file said to look locally ("call home") and since there was a computer with a "heart beat", the timeout took less than 1 second.  The fact that your computer CAN NOT perform Certificate checking on these plug-ins and their files is irrelevant.  There are some big downsides of doing this:

  1. This is a manual process to create and maintain these entries.
  2. There is a good chance that these websites addresses will change with future releases of their products (I know that one of Dragon Naturally Speaking crl sites changed between v10 and v12)).
  3. Since I can't use wildcards there is a good chance that I will miss one of the Certificate sites that the plug-in may reference later on in the usage of the application.
  4. OH YEA, this is just a horrible idea (did I mention that I felt dirty even trying this?)!

So, what is a better way to handle this?  Remember the modification of the HOSTS. file should only be used for testing or a temporary solution.  Also remember this is a CLN (Closed Loop Network), any traffic destined for ANYTHING.Microsoft.com or ANYTHING.Symantec.com or ANYTHING.Verisgn.com will NEVER make it.  Time to break out the BIG GUNS and start creating zones in DNS.

However I will cover what we tested and finally did in DNS tomorrow.

………… to be continued.

Comments

  • Anonymous
    March 19, 2017
    I've had this post open in a browser tab for over a week waiting for the "to be continued". Any news on when it's coming? :)
    • Anonymous
      March 29, 2017
      Our apologies. Bill will be updating the blog soon.
      • Anonymous
        June 27, 2017
        Better get Bill back again :) Another "Tomorrow..." finish. At least I suspect the fix here is fairly straightforward (sinkhole things to 127.0.0.1 at a broad scale)
  • Anonymous
    March 28, 2017
    Excellent Post. Very informative and insightful.