Outlook 2010 - network latency test results

Over the past 18 months I have been dealing with more and more cloud engagements.  One thing that crops up during the planning phase of all of these engagements is network link latency and bandwidth requirements.

Bandwidth prediction is something I am working on in the background, but network link latency seemed like a mystery!.  Everywhere I looked and everyone I spoke to had a different opinion on “maximum latency” for Outlook clients.

So.. I decided to try to devise a test that would let me model the impact of network link latency on Outlook client performance.  My lab already had a network emulator installed that was capable of simulating link latency so all that I needed was a script to automate Outlook to perform some common tasks, then record how long the script took to run – easy right?

Automating Outlook…

I already had some scripts for Outlook 2010 from my bandwidth prediction work, so I decided to make use of them.  The scripts were written with AutoIT which is a regular part of my lab testing toolkit these days!

Software versions in use for all tests…

• Windows 7 Service Pack 1 32-bit
• Outlook 2010 Service Pack 1
• Exchange Server 2010 Service Pack 1

Note: All tests began from a cold start i.e Outlook was not running.

Test definitions

Note: All clients were tested from 0ms added latency to 1000ms in steps of 100ms and then up to failure point from 1000ms onwards in steps of 500ms. Failure was deemed to be if any single test failed.

Test 1. Sending

• Send 5 messages with 2 x 50kb attachments

Test 2. Opening

• Open 10 messages with 2 x 50kb attachments
• save attachments to disk

Test 3. Delegates

• Open delegates
• Add in a new delegate
• Remove delegate

Test 4. Out of Office

• Open out-of-office
• Enabled message and set OOF
• Switch OOF off

Test 5. Calendar

• Open calendar
• Add in shared calendar
• close shared calendar

 

Notes:

• For all cached mode tests a final sync was performed and test did not completed until the outbox was empty.
• Outlook Anywhere was configured to connect over HTTPS on fast networks to reduce start-up delays.
• Windows 2008 R2 CA PKI was used to issue Exchange certificates in the lab.

Results

I decided to run tests 1-5 sequentially for three types of client connections

• Outlook 2010 Online
• Outlook 2010 Cached Mode
• Outlook 2010 Outlook Anywhere (RPC/HTTPS)

Note: All latency values represented are round-trip times RTT as reported by ping.exe from a command line.

Combined test case results

What is evident from this set of results is that the impact of network link latency is linearly related to outlook client performance.  Certainly when we consider a broad range of functions.  Cached mode dramatically reduces the user wait time for send/receive functions, but it was obvious watching the test run that whenever actions were requested that ran outside of the OST file, such as delegate access or out-of-office requests large delays were experienced at higher latency values.  I decided to set the maximum test run time limit at 200 seconds, which was double the test run time with no additional latency applied.  Client behaviour when the test duration was taking 200 seconds was the poorest performance level that I perceived to be acceptable by anyone for this test.  I judged that if any single operation took 10 seconds or more to display a dialog box or the client appeared to hang then it was unacceptable for the end user. 

In this combined test the limiting task was working with the delegate permissions, which was taking approximately 9 seconds to complete at the maximum test run time.  Accessing delegate permissions was similar amongst all connection modes. 

The results for Outlook Anywhere were confusing initially, although the majority of the difference between Cached and Cached-OA is due to the additional time it takes for an Outlook Anywhere connected client to establish its initial connection.  Typically an OA client will take around 10 seconds longer to establish a connection when compared to a normal RPC MAPI connection.  This rises to around 30 seconds at 800ms latency.  Under use both Outlook Anywhere and Cached mode performed with very similar results.

Combined test conclusion

The test results for me were quite interesting.  Prior to this testing I had assumed that Outlook tolerance for link latency was was much worse than the test results show.  This is no doubt due to changes in adaptive TCP windowing on Windows 7 plus improvements in Outlook 2010.  The bottom line is clear though, Outlook 2010 on Windows 7 is able to tolerate quite high network link latency values before performance begins to suffer noticeably.  In the case of Cached mode and Outlook Anywhere both clients were able to tolerate 500ms round-trip-time latency without problem and if your main tasks are sending and receiving mail only, cached mode actually provided a good experience at up to 1000ms round-trip-time latency.  Online mode was far more sensitive to network link latency as expected and performance became noticeably slower at 190ms round-trip-time and above.  Online mode was still usable above 200ms latency but felt sluggish.

Failures

One thing that I didn't account for during this test was that the clients would behave differently as latency increase.  My automated script initially didn't cope very well when Outlook in online mode was subjected to high latency values.  The long pauses before dialog boxes were displayed made things quite tricky at times, AutoIT is extremely capable though which really helped in this testing!.

The following failure points were noted..

• Outlook 2010 Online Mode – General script failure at 800ms (gave in trying to make it work – “Outlook is not responding” )
• Outlook 2010 RPC/HTTPS – Delegates page failed to open at 2000ms
• Outlook 2010 Cached Mode – Delegates page failed to open at 2000ms

Note: Although the cached mode clients failed at 2000ms for delegate access the client continued to provide usable mail performance.  I plan to run further tests in cached mode to see where the break point is for mail only users since this represents the majority of users.  Keep watching my blog for more information on this test

Other thoughts…

This test was designed to isolate the effects of network link latency on client performance, however the reality is that network link latency is only one part of the Outlook client performance experience.  In practice it is more likely that poor client performance is caused by local hardware limitations or insufficient resources allocated on the Exchange Server itself.   This testing does show however, that consolidating Exchange Server resources into central locations is viable with Windows 7 and Outlook 2010.

OK, that's great, but what about OST file resync?

So.. this is an interesting question that came to mind while I was performing this testing.  We can provide a good user experience in normal day to day tasks over quite latent network links, but what happens if I need to rebuild my OST file?  OST file resync is a fairly common occurrence; even if we just consider cases of SATA HDD failure (5% AFR typically), that means in an organisation of 30,000 laptop users there will be approximately 1500 OST resync’s per year (7.5 per working day!) due to HDD failure.  I was curious about how long it would take to synchronise an OST file under specific latency conditions, so I ran some tests….

I created a 100MB mailbox then timed how long that took to synchronise down as latency increased, I then calculated how many seconds it took per 1MB of data.  My plan was to use this to extrapolate how long larger mailboxes would take to synchronise down.

Note: I had initially planned on doing this experimentally but then the reality of having to run 10 or more tests each taking 20 hours or longer to complete set in and I decided to deal with it theoretically. I may run a few tests to validate the theory when i get time though.

Using this data I was able to extrapolate how long it would take to fully synchronise down a 25GB mailbox (default Office 365 quota limit).  Given the assumption that there is always sufficient network bandwidth available, the data looks as follows…

This data suggests that even if a cached mode client were running over a  500ms round-trip-time network latency connection that it would be possible to fully resynchronise their 25GB OST file within 43 Hours.  This would require a throughput of 1.34Mbits/sec.

One of the customers I am working with is considering a 5GB quota within their Office 365 tenant so I promised them I would consider their case specifically, so here is the data for a 5GB OST sync.

Taking the same point of 500ms, this prediction suggests that their 5GB OST file would be fully synchronised after 8.6 Hours, assuming that the client had 1.33Mbits/sec of network bandwidth available.

To perform your own predictions using this data use the following form…

T = ( M ) x (( 0.0114 x L ) + 0.3208 )

Where

• T = Time to Sync ( seconds)
• M =Mailbox Size (Megabytes)
• L = Network Link Latency Round-trip-time (milliseconds)

 

Further tests

This testing sparked a few thoughts in my mind, that I just didn't have time to follow up (and I was being harassed by various colleagues and customers to release the data i had!) … anyway, the following are areas i plan to research further when time allows…

• What effect does disabling the Windows 7 TCP Adaptive Windowing feature have on Outlook client performance over latent network links?
• How does Outlook 2007 compare to Outlook 2010 in this test?
• Where do Cached Mode and OA clients break when just sending and receiving E-mail

 

Final Conclusion…

For me this test data suggests that network latency is not as critical for client performance on Windows 7 with Outlook 2010 cached mode as it used to be on legacy operating systems.  It also shows that the adaptive TCP windowing in Windows 7 goes some way to alleviate the negative effects of network latency.   Certainly for my current projects it has allowed me to consolidate more mailboxes into a central location than i would have initially thought possible and the client involved tells me that end user performance is good.

If you are considering running Outlook clients over high latency connections I would urge you to perform a full user pilot to establish your users tolerance of client performance.  Also some Outlook plugins can have a dramatic effect on how the client performs under very latent conditions.

Neil.

Updated: 17/06/2011

Comments

• Anonymous
  January 01, 2003
  Hi Neil,
  
  I just bumped into your presentation about Exchange Client Network Bandwidth Calculator v2.
  In April 2014 you mentioned that the tool will come out very soon... Any idea when we can expect this :)
  
  Good article !

• Anonymous
  January 01, 2003
  I am happy to share, but to be honest they are not much use since I resorted to using mouseclick() for most things since it seemed more reliable than keyboard shortcuts when the client started to run slowly... and I didnt have Outlook maximised so the scripts are very tightly integrated to a single test machine in my lab! :( Its on my "to do" list to repeat this on Outlook 2007 though...  hopefully this week

• Anonymous
  January 01, 2003
  I too am curious about this and for legacy operating systems.  If I find time I will definitely look to analyse some legacy technology also... the trouble with this kind of testing is that it takes lots of lab time to prepare and perform the testing.  I usually try to fit this stuff in between customer engagements, but we are frantically busy at the moment so unless an engagement I am working on needs the data its difficult to make the time to do it.. still its definitely something i want to analyse further when i get the time.

• Anonymous
  January 01, 2003
  Very Nice Job Neil :)

• Anonymous
  January 01, 2003
  Terrific article, Neil!  Thanks for sharing your test results. I'd be curious to know what the results look like for Outlook 2007, since I run into quite a few clients who have not upgraded yet and are looking for a business reason to do so.

• Anonymous
  June 17, 2011
  Hi, for your info, I did some tests before we migrated to exch2010 and compared Outlook2007 and 2010. And for me, 1gb mailbox took about 15min to sync on 2010 (with ~ 45ms latency) compared to 45min on Outlook 2007 (both in cache mode).

• Anonymous
  June 17, 2011
  Great post Neil, thanks for putting in the hard work needed for this and doing the write up!

• Anonymous
  June 25, 2011
  This is excellent,  Would you be able to share you scripts?  Most of the estate I am in at the moment in still XP and Outlook 2007.  It would be good to try and do a comparison

• Anonymous
  July 11, 2011
  Excellent Article! Thanks for sharing this Neil!

• Anonymous
  April 18, 2012
  The comment has been removed

• Anonymous
  September 18, 2012
  Hi Neil, great article!  Is there by chance a test conducted against O365 (MS cloud exchange) with the same parameters?  Would you consider there to be any variance of your findings above with anything unique with respect to O365? We're trying to determine how to optimize our O365 deployment and would like to use your study as a benchmark.  Thanks!