How To: Identify Memory Leaks In An Unmanaged Application

Syed Aslam Basha here. I am a tester on the Information Security Tools team.

To carryout memory leaks testing for one of our applications, I have researched a lot and I thought I would share my experiences and approach I used to benefit everybody.

Application details:

My application under test is a DLL which hosts in winlogon.exe. We are building credential provider.

Collecting data through perfmon and analyzing results:

I will refer my machine as server( which has Vista OS) where the application is installed and Client(which has windows XP) from where the application is being stressed.

1) Login in server

2) Start perfmon – (server)

3) Login in client

4) Do mstsc to server

5) Set the counters(private bytes, virtual bytes, # Bytes in heaps, Handle count, processor) in server perfmon  – (server)

6) Stress the application from client say for 2 hours. This will be done by an automated script – (client)

7) Stop sending requests to server

8) Wait till 15 minutes

9) Stop perflog – (server)

10) Analyze perfmon logs and find out the type of leaks and instrument debugdiag (one of memory leaks testing tools) to monitor those counters

The above steps will be repeated once with application under test and without application under test to determine the delta of memory leaks

Setup Debugdiag to monitor memory leaks:

Some of the ways to Setup Debugdiag to monitor memory leaks:

Approach A)Manual

1) Login in my machine

2) Start debugdiag

3) Start perfmon - (configure counters to monitor for the winlogon process)

4) Recycle the leaky process - (this is not possible)

5) Attach DebugDiag (either manually or by a Leak Rule) – (create leak rule)

6) Stress the application. (This will be done manually, I will lock my machine and stress the login prompt manually )

7) Reproduce the leak.  Allow the leak to occur for the adequate length - ( say 4 hours)

8) After sufficient time has passed, use DebugDiag to dump the process.

9) Stop sending requests to the leaky process

10) Wait 15 minutes or so, and then dump the process again. (This step is optional)

11)  Stop the Perfmon log

12) Analyze the perfmon log

13) Analyze the dump taken in step 9

Approach B)Using TS and auto stress the application

Server will have my application and debugdiag

Client will have automated script to stress the application through mstsc

1) TS in to server from client machine

2) Start debugdiag

3) Start perfmon - (configure counters to monitor for the winlogon process)

4) Recycle the leaky process - (this is not possible)

5) Attach DebugDiag (either manually or by a Leak Rule) – (create leak rule)

6) Lock the TS session

7) Stress the application – (client)

8) Reproduce the leak.  Allow the leak to occur for the adequate length - ( say 4 hours)

9) After sufficient time has passed, stop sending requests to the leaky process –  (client)

10) TS in server use DebugDiag to dump the process.

11) Stop the Perfmon log

12) Analyze the perflog

13) Analyze the dump taken in step 10

Thumb rule: If Private Bytes increases over time while #Bytes In All Heaps remains relatively flat, then the leak is a native one.

- Syed