Slow performance when accessing a Sharepoint 2010 portal through an ISA 2006 array
I was involved in a slow Sharepoint portal access through an ISA 2006 array problem and I wanted to talk about how I dealt with that issue and found the root cause of the slowness issue. The problem was that the portal home page was loaded very slowly and sometimes it wasn’ loaded at all from external clients. The performance was fine when it was accessed from internal clients
Our customer had a network setup similar to below:
We collected network traces from ISA servers while reproducing the problem. Network trace analysis revealed that ISA servers were responding to the requests immediately but there were delays in the incoming HTTP requests from the external clients and in some occasions external client didn’t complete the TCP three way handshake:
=> Examples of such huge gaps between requests from the external client seen in the network trace:
Note: 1.1.1.1 is the external client (real IP address was replaced)
Note: 192.168.1.36 is the web listener IP address to which sharepoint publishing rule is bound
17324 14:11:33.045375 11.859375 0.015625 1.1.1.1 192.168.1.36 HTTP GET /UserControls/TopMenu/TopMenu_Sep.gif HTTP/1.1
17326 14:11:33.045375 11.859375 0.000000 192.168.1.36 1.1.1.1 HTTP HTTP/1.1 304 Not Modified
17522 14:11:33.232875 12.046875 0.187500 1.1.1.1 192.168.1.36 HTTP GET /_layouts/1033/ie66up.js?rev=Ni7%2Fj2ZvA%33D HTTP/1.1
17524 14:11:33.232875 12.046875 0.000000 192.168.1.36 1.1.1.1 HTTP HTTP/1.1 304 Not Modified
17529 14:11:33.248500 12.062500 0.015625 1.1.1.1 192.168.1.36 TCP 1273 > 80 [ACK] Seq=2525698284 Ack=1206119216 Win=65664 Len=0
17533 14:11:33.264125 12.078125 0.015625 1.1.1.1 192.168.1.36 TCP 1269 > 80 [ACK] Seq=640090889 Ack=3984935419 Win=65340 Len=0
17593 14:11:33.451625 12.265625 0.187500 1.1.1.1 192.168.1.36 TCP 1268 > 80 [ACK] Seq=1670250900 Ack=2758693123 Win=65576 Len=0
There’s a huge time gap (around 83 seconds) in between the last response from the ISA server and the next request from the external client:
153758 14:12:56.998500 95.812500 83.546875 1.1.1.1 192.168.1.36 TCP 1268 > 80 [FIN, ACK] Seq=1670250900 Ack=2758693123 Win=65576 Len=0
153759 14:12:56.998500 95.812500 0.000000 192.168.1.36 1.1.1.1 TCP 80 > 1268 [ACK] Seq=2758693123 Ack=1670250901 Win=64684 [TCP CHECKSUM INCORRECT] Len=0
153760 14:12:56.998500 95.812500 0.000000 1.1.1.1 192.168.1.36 TCP 1269 > 80 [FIN, ACK] Seq=640090889 Ack=3984935419 Win=65340 Len=0
153761 14:12:56.998500 95.812500 0.000000 192.168.1.36 1.1.1.1 TCP 80 > 1269 [ACK] Seq=3984935419 Ack=640090890 Win=65535 [TCP CHECKSUM INCORRECT] Len=0
153762 14:12:56.998500 95.812500 0.000000 192.168.1.36 1.1.1.1 TCP 80 > 1268 [FIN, ACK] Seq=2758693123 Ack=1670250901 Win=64684 [TCP CHECKSUM INCORRECT] Len=0
...
158541 14:13:06.639125 105.453125 0.015625 192.168.1.36 1.1.1.1 HTTP HTTP/1.1 304 NOT MODIFIED
158576 14:13:06.842250 105.656250 0.203125 1.1.1.1 192.168.1.36 TCP 1290 > 80 [ACK] Seq=2442652609 Ack=3279316510 Win=65376 Len=0
Another time gap which is around 11 seconds in between the last response from the ISA server and the next request from the external client:
178031 14:13:18.014125 116.828125 11.171875 1.1.1.1 192.168.1.36 TCP 1292 > 80 [SYN] Seq=2506205172 Win=8192 Len=0 MSS=1380 WS=4 SACK_PERM=1
178032 14:13:18.014125 116.828125 0.000000 192.168.1.36 1.1.1.1 TCP 80 > 1292 [SYN, ACK] Seq=1749486699 Ack=2506205173 Win=16384 Len=0 MSS=1460 WS=1 SACK_PERM=1
178039 14:13:18.029750 116.843750 0.015625 1.1.1.1 192.168.1.36 TCP 1292 > 80 [ACK] Seq=2506205173 Ack=1749486700 Win=66240 Len=0
178042 14:13:18.029750 116.843750 0.000000 1.1.1.1 192.168.1.36 HTTP GET /Style%20Library/Images/leftCol_02.gif HTTP/1.1
178043 14:13:18.029750 116.843750 0.000000 192.168.1.36 1.1.1.1 HTTP HTTP/1.1 304 NOT MODIFIED
178202 14:13:18.248500 117.062500 0.218750 1.1.1.1 192.168.1.36 TCP 1292 > 80 [ACK] Seq=2506206023 Ack=1749486987 Win=65952 Len=0
255223 14:13:58.092250 156.906250 39.843750 1.1.1.1 192.168.1.36 TCP 1289 > 80 [RST, ACK] Seq=3712234005 Ack=2023647270 Win=0 Len=0
=> Another indication of a connectivity problem between the external client and the ISA 2006 server:
No. Time Time since Delta Source Destination Protocol Info
153775 14:12:57.014125 95.828125 0.000000 1.1.1.1 192.168.1.36 TCP 1288 > 80 [SYN] Seq=2013470292 Win=64860 Len=0 MSS=1380
153776 14:12:57.014125 95.828125 0.000000 192.168.1.36 1.1.1.1 TCP 80 > 1288 [SYN, ACK] Seq=535425124 Ack=2013470293 Win=16384 Len=0 MSS=1460
154809 14:12:59.342250 98.156250 2.328125 192.168.1.36 1.1.1.1 TCP 80 > 1288 [SYN, ACK] Seq=535425124 Ack=2013470293 Win=16384 Len=0 MSS=1460
158232 14:13:05.904750 104.718750 6.562500 192.168.1.36 1.1.1.1 TCP 80 > 1288 [SYN, ACK] Seq=535425124 Ack=2013470293 Win=16384 Len=0 MSS=1460
ISA receives TCP SYN from the external client and sends back a TCP SYN ACK immediately. Normally the client should have continued with a TCP ACK to finish the TCP 3-way handshake and then send the HTTP request on top of that TCP session but that doesn’t happen and ISA 2006 re-sends the TCP SYN ACK and resends one more time. For example this activity alone causes some 10 seconds to be lost.
Just to eliminate any network device issues we decided to access the same sharepoint portal just from the front of ISA array’s external interface (192.168.1.32/27 subnet) and the client was assigned an IP address from that subnet. After some testing from that client it was possible to quickly access the portal (and the test machine has no difference from an external client out in the internet from ISA perspective), it confirmed that the problem is not with ISA array but it’s a problem somewhere between ISA array’s external interface and our customer’s internet connection including the Cisco ASA device/internet routers.
Our customer’s network team was involved in and after the issue with the router/firewall was fixed, external clients also started accessing the portal very quickly.
Hope this helps
Thanks,
Murat