Issues with Perfmon reporting - Turning ETL into HTML

Hi Everyone.

It's Marc here. Another of your friendly Canberra PFE's.

Recently I came across a problem with Performance Monitor's reporting functionality in Windows Server 2008 R2. More specifically, reporting with the Active Directory Diagnostics template.
A customer was having a hard time with their Domain Controller CPU levels being sustained at or above 90% - initial investigation with our old friend 'Task Manager' showed this was LSASS.exe. To add to this, there were no baselines for us to check to see if this was a growing trend, or if this was just a random spike over the last few days.

The best place to start when troubleshooting high LSASS.exe CPU usage is detailed here > https://blogs.technet.com/b/askds/archive/2007/08/20/troubleshooting-high-lsass-cpu-utilization-on-a-domain-controller-part-1-of-2.aspx. I am not going to cover this, because Ned already smashes that topic out of the park, and after all, this is a post about issues with reporting in perfmon...

So, back on topic. I suggested for them to run the 'Active Directory Diagnostics' Data Collector Set.

After the customer had created a Data Collector Set from a template (https://technet.microsoft.com/en-us/library/cc766318.aspx). It was setup to run for what we thought was going to be a quick 5 minutes to try and do a quick and dirty diagnostic. This 5 minutes turned slowly into hours as Performance Monitor tried to create  the report with the little CPU time that it was able to get.

The next day, I asked the customer if they could send me the Report.html file. I was quickly told - there is no report.html.  Hmmmm… Could there be a scheduled action which has cleaned this up? By default, reports older than 24 weeks will be deleted. After a quick look, it was discovered that this was this case - the report was less than 1 day old.  You can check the rules on your server by right clicking on the DCS and selecting 'Data Manager' > 'Actions' Tab.

 

All they had for me were the following files;

This was a 300 second trace, and the ETL files totaled over 1GB! No wonder this took so long - there is a lot of data.

File Size	File Name
809,828,352	Active Directory.etl
7,284	AD Registry.xml
199,426,048	NtKernel.etl
7,733,248	Performance Counter.blg
62,745	report.xsl
12,047	summary.txt

 

So, all I had were the associated files, and all I wanted was a nice performance report to send back to the customer without me having to interpret this data manually - after all it is not really practical with ~1GB worth of ETL files. So thanks to a bit of searching I found this article.

The report generation process may stop responding when you run Perfmon.exe with the Active Directory Diagnostics template to generate a report on a Windows Server 2008-based domain controller.
https://support.microsoft.com/kb/971714

Step 4;
Locate the files, and then run the following command at the command line on the second Windows Server 2008-based domain controller:

tracerpt *.blg *.etl -df your_tmp_filename.tmp -report your_report.html -f html

Looking at the command in step 4, it asks for a .tmp file I don’t have a .tmp file… Yes, I know the DC wasn’t freezing when generating the report, it just didn’t do it - and I didn’t have time to rerun the DCS and find out why.

So, what next you ask? You get all the way here and the above KB is useless…? That’s what I was thinking when I got to this point.

I jumped on another Windows Server 2008 R2 server and kicked off another DCS based on the 'Active Directory Diagnostics' template. I looked in the output directory of the DCS and Bingo! There is "RPTE290.tmp"!! I copied this out while the collector was running and fed this .tmp file into the command above, and about 15 hours later I had the report I was after. Did it have the answers to all my problems? Not this time. So, after all this I had an answer for the customer, I just wasn’t able to pin point the issue this time with 5 minutes worth of data. Hopefully, this will be helpful and the next time someone needs to do this the report has the answer they are looking for.

Until next time.

Marc "Need a cool quip like Ned Pyle" Dudok.