Distributed Architecture Drawbacks Revealed By Netmon(Bonus - TDS Parser Goes Public)

Distributed architecture can mercilessly backfire at you.  In my case flexible architecture, elegant design patterns, and smart code led to abuse of the flexibility, resulting in very poor performance. Free Microsoft Network Monitor (Netmon) helped to identify the root cause of the

performance hit. It showed that over-distributed-ness can cost you in terms of performance. Customer Case Study The customer complained about poor response times in his web application. The application's architecture was similar to the Web Application Archetype. Notice Services Agent box that connects your application to downstream services? Our assumption was that the services agents are too chatty causing the performance hit. Netmon only made it clear. Analysis We took captures on the application server where the Server Agents are to identify what other downstream servers are accessed. In no time we get very clear picture - the application server was accessing other three downstream resources:   There are two well known ports - 443 and 1433 - so we could safely assume there is communication over SSL/HTTPS and SQL Server. The other one - 1414 - turned out to be MQ. Next step was to identify which one of the conversations is causing us troubles the most - either by chatty communication or by just long running transaction. Looking at Time Delta column for HTTPS stateless communications we found nothing exciting regarding the latency: For MQ communications we used magic ContainsBin(FrameData, 0,"StringToFindGoesHere") to identify XML messages going back and forth over MQ transport. For example, to find the beginning of the XML message we used the following filter: Similar technique was used to correlate request and response XML messages. To identify SQL communication we used shiny new TDS parser available for free download on Codeplex - SQL Parser in Latest CodePlex Package. I particularly like this one, it shows SQL Server communication without using SQL Server Profiler:   Conclusion Lessons learned: Distributed Architecture can mercilessly backfire at you by its distributed-ness. Use Distributed architecture to solve problems vs introducing new ones. Elegant and flexible design can be abused resulting in poor performance. Review code to avoid the abuse. Low level network monitoring tools such as Netmon can quickly reveal over-distributed-ness. Use it! It is free. Related Materials Do You Really Need A Distributed Architecture? ASP.NET Performance Engineering - Stress Test Your Architecture, Design, And Code

This post is made with PracticeThis.com plugin for Windows Live Writer

Comments

• Anonymous
  March 13, 2009
  Alik, this is an excellent case study which documents not only fundamentals of analysis, diagnosis, & remediation, but also & perhaps more importantly that we oughtn't jump on the latest-&-greatest simply because either 1) marketeering hype &/or 2) that the latest-&-greatest often isn't actually the greatest.
• Anonymous
  March 13, 2009
  The comment has been removed