KB942864 (Update) - How to use diagnostic tracing in System Center Operations Manager 2007 and in System Center Essentials 2007
I stated in a previous post that the existing trace KB article was undergoing a refresh due to changes in OpsMgr SP1 and SCE SP1. The updated article is now available https://support.microsoft.com/kb/942864
Details below:
INTRODUCTION
This article describes how to use diagnostic tracing in Microsoft System Center Operations Manager 2007 and in Microsoft System Center Essentials 2007.
Important We recommend that you perform diagnostic tracing only in association with a Microsoft Customer Support Services (CSS) representative. We recommend this because the generated traces contain information about the context of a text-based trace message. However, in Microsoft System Center Operations Manager 2007 and in Microsoft System Center Essentials 2007, this trace information is not in a human-readable format. After the traces are converted by a Microsoft CSS representative, human-readable text is available. However, this text contains only low-level information such as source-code file names, locations, source-code functions, and return codes. This information may be helpful if you have to troubleshoot a complex issue.
In System Center Essentials 2007 Service Pack 1 (SP1), in System Center Operations Manager 2007 SP1, and in later versions, trace message files (TMF) are supplied which allow for conversion of binary trace files to text. However, we strongly recommend that you perform diagnostic tracing only when it is required and only with the consultation of a Microsoft representative.
MORE INFORMATION
System Center Operations Manager 2007 and System Center Essentials 2007 implement a diagnostic tracing method that differs from earlier versions of Microsoft Operations Manager. This new tracing method creates binary files in which to store tracing information. Because this new tracing method is implemented at the Windows kernel level, it is highly efficient, and it can log tens of thousands of trace messages per second.
The following information discusses the tools that are available to start and to stop tracing if you are asked to do this by a Microsoft CSS representative. The following information also discusses the new functionality that is provided in System Center Essentials 2007 SP1, in System Center Operations Manager 2007 SP1, and in later versions.
Tracing tools location
The installation folder for each role, such as the Agent role, the Management Server role, or the Gateway role, contains a folder that is named Tools. The following files are located in the Tools folder:
•
StartTracing.cmd
•
StopTracing.cmd
•
TracelogSM.exe
•
TracingGuidsBid.txt
•
TracingGuidsUI.txt
•
TracingGuidsNative.txt
•
TracingReadMe.txt
System Center Essentials 2007 SP1 and System Center Operations Manager 2007 SP1 introduce the following additions:
•
FormatTracing.cmd
•
ViewRealtimeTracing.cmd
•
TraceFmtSM.exe
•
OpsMgrTraceTMF.cab
•
Default.tmf
•
System.tmf
To start tracing
Note In System Center Essentials 2007 SP1, in System Center Operations Manager 2007 SP1, and in later versions, tracing is automatically started on both the Agent role and the Management Server role. Tracing will use error output only. Binary trace files are written to the SystemDrive\Temp\OpsMgrTrace folder. Before the tracing level for trace output can be changed, tracing must first be stopped.
To start diagnostic tracing, follow these steps:
1.
On the computer on which you want to start tracing, click Start, click Run, type cmd, and then click OK.
2.
At the command prompt, use the cd command to change to the Tools directory. For example, type the following command, and then press ENTER:
cd\Program Files\System Center Operations Manager 2007\Tools
3.
Type StartTracing LEVEL, and then press ENTER. In this command, replace LEVEL with the tracing level that you want. You must use uppercase characters to specify the tracing level. The following levels are available:
•
ERR
•
WRN
•
INF
•
VER
For example, type StartTracing WRN.
When you start tracing, the trace output is written to the following binary files:
•
MOMTraceNative.etl
•
MOMTraceBID.etl
•
MOMTraceUI.etl
These files are located in the SystemDrive\Temp folder.
Note In System Center Essentials 2007 SP1, in System Center Operations Manager 2007 SP1, and in later versions, trace files (.etl) are located in the SystemDrive\Temp\OpsMgrTrace folder.
To stop tracing
To stop diagnostic tracing, follow these steps:
1.
Click Start, click Run, type cmd, and then click OK.
2.
At the command prompt, use the cd command to change to the Tools directory. For example, type the following command, and then press ENTER:
cd\Program Files\System Center Operations Manager 2007\Tools
3.
Type StopTracing.cmd, and then press ENTER.
After you stop diagnostic tracing, you can send the .etl files from the SystemDrive\Temp folder to the Microsoft CSS representative with whom you are working.
Note In System Center Essentials 2007 SP1, in System Center Operations Manager 2007 SP1, and in later versions, trace files are located in the SystemDrive\Temp\OpsMgrTrace folder. Because TMFs are supplied with SP1, traces can be converted without having to send .etl files to Microsoft.
To convert existing traces to text format in System Center Essentials 2007 SP1, in System Center Operations Manager 2007 SP1, and in later versions
To convert existing trace files to text tracing, you have to stop tracing first by using the StopTracing.cmd command.
Note Information within the converted traces is of a very low level detail (debug level). Therefore, the converted traces require knowledge of the source code and of the component that is being traced. We recommend that trace conversion be performed only when it is required by a Microsoft CSS representative.
1.
Click Start, click Run, type cmd, and then click OK.
2.
At the command prompt, use the cd command to change to the Tools directory. For example, type the following command, and then press ENTER:
cd\Program Files\System Center Operations Manager 2007\Tools
3.
Type FormatTracing.cmd, and then press ENTER.
The first time that the FormatTracing.cmd command is executed, the trace files that are required to convert traces are extracted to the SystemDrive\Program Files\System Center Operations Manager\Tools\TMF folder. Then, the FormatTracing.cmd batch file enumerates each trace file (*.etl) in the SystemDrive\Temp\OpsMgrTrace folder. Then, the TraceFMTSM.exe utility converts the files to text. The text output is written to a file of the same name with the extension .log in the SystemDrive\Temp\OpsMgrTrace folder. A summary (.sum) file is also written to the same location for each file that is converted. This file details each trace message event converted.
Viewing converted trace files in System Center Essentials 2007 SP1, in System Center Operations Manager 2007 SP1, and in later versions
Trace files that are converted to text by using the FormatTracing.cmd batch file can be viewed by using a text editor, such as Notepad. By default, in System Center Essentials 2007 SP1, in System Center Operations Manager 2007 SP1, and in later versions, trace files are located in the SystemDrive\Temp\OpsMgrTrace folder.
Boot time tracing is enabled by default in System Center Essentials 2007 SP1, in System Center Operations Manager 2007 SP1, and in later versions
On a typical Root Management Server (RMS) installation, three trace sessions begin when the HealthService service is started. By default, only error tracing is performed. Little information is written to the trace files. The default trace file location and names that are created are as follows:
Folder
File name
Description
system drive\Temp\OpsMgrTrace
TracingGuidsBID.etl
Trace output for Managed Code Components
system drive\Temp\OpsMgrTrace
TracingGuidsNative.etl
Trace output for Native Code Components
system drive\Temp\OpsMgrTrace
TracingGuidsUI.etl
Trace output for Managed code User Interface (OpsMgr UI)
Each session is enabled with circular tracing and has a maximum file size of 100 megabytes (MB). The typical .etl file size for a newly created .etl file is 16 kilobytes (KB).
Note Although an Operations Manager Role, such as an Agent role, does not contain managed or user interface tracing messages, the three default trace .etl files will be created by default when the HealthService service is started. By default, no trace sessions are started on a User Interface only role.
How to disable boot time tracing in System Center Essentials 2007 SP1 and in System Center Operations Manager 2007 SP1
Warning Serious problems might occur if you modify the registry incorrectly by using Registry Editor or by using another method. These problems might require that you reinstall the operating system. Microsoft cannot guarantee that these problems can be solved. Modify the registry at your own risk.
Three trace providers are started automatically on any role that contains the HealthService service. The trace files are written to the SystemDrive\Temp\OpsmgrTrace folder. Typically, the .etl files in this folder will be small, especially for the Agent role. However, for the RMS role, if the operations manager installation becomes unhealthy, one or more of these files could potentially grow to the maximum configured size of 100 MB each. These files include the following:
•
TracingGuidsBid.etl
•
TracingGuidsNative.etl
•
TracingGuidsUI.etl
•
previous .etl files
If you have to disable debug level tracing because of a small boot partition size, you can add a registry value to the computer that is running the Management Server role or the Agent role on which you want to disable tracing. To do this, follow these steps.
Note If tracing is disabled, this may affect future troubleshooting of Operations Manager because any low level tracing information will be lost. Future hotfixes, service packs, or product updates may remove the registry entry and enable tracing again. By default, this registry key does not exist. Default tracing is enabled if you remove this registry key or change its value to 0.
1.
Click Start, click Run, type regedit, and then click OK.
2.
Locate and then click the following registry subkey:
HKEY_LOCAL_MACHINE\Software\Microsoft\Microsoft Operations Manager\3.0\Tracing
3.
In the details pane, double-click DisableAutoTracing.
4.
In the Value data field, type 1, and then click OK.
5.
Exit Registry Editor.
If you execute the StopTracing.cmd batch file from the Tools folder, the Operations Manager trace sessions will be stopped. The .etl files in the SystemDrive\Temp\OpsmgrTrace folder can be deleted if they are no longer required to regain disk space.
How to move the default location of trace file output in System Center Essentials 2007 SP1, in System Center Operations Manager 2007 SP1, and in later versions
If an Operations Manager Server role or Agent role is installed on a computer where the boot partition space is limited, you may have to move the default location of trace files (*.etl). To do this, follow these steps:
1.
Create a new folder on a local disk that has sufficient capacity to house the files. Three default trace files at a maximum of 100 MB each and three previously-used trace files at a maximum of 100 MB each equals a total potential trace file capacity of 600 MB.
2.
Set at least the following NTFS permissions on the folder, SYSTEM = Full Control, Administrators = Full Control.
3.
Locate and open the StartTracing.cmd batch file in the Operations Manager/Essentials/Tools installation folder. Use a text editor, such as Notepad.
4.
In the StartTracing.cmd batch file, locate the SET OpsMgrTracePath statement. Replace the default value SystemDrive\Temp\OpsMgrTrace with the path of the new location. Delimit the path with speech marks (“) if the path contains a space character.
5.
Save the changes to the StartTracing.cmd file.
6.
To start tracing to the new folder, run the StopTracing.cmd batch file. Then, run the StartTracing.cmd batch file. After you run the StartTracing.cmd batch file, the following files are written to the new folder:
•
TracingGuidsBid.etl
•
TracingGuidsNative.etl
•
TracingGuidsUI.etl
7.
Before you use the FormatTracing.cmd command to convert traces to text, you must edit the FormatTracing.cmd command to change the OpsMgrTracePath variable to point to the new trace file folder.
Future hotfixes, service packs, or product updates may change the files in the Tools folder. This changes the functionality back to the default functionality. We recommend that you check for more changes after you perform updates.
How to view real time tracing in System Center Essentials 2007 SP1, in System Center Operations Manager 2007 SP1, and in later versions
We recommend that you do not use the ViewRealTimeTracing.cmd batch file to view real time traces in a production environment without advice from Microsoft CSS. Viewing traces in real time, especially when you use Information trace level settings or Verbose trace level settings, may severely affect server performance.
The first time that the ViewRealTimeTracing.cmd file is executed, the trace files that are required to convert traces are extracted to the SystemDrive\Program Files\System Center Operations Manager\Tools\TMF folder. The ViewRealTimeTracing.cmd batch file then enumerates each trace session definition file (Tools\TracingGuids*.txt) in the Tools folder. Then, a separate instance of TraceFmtSM is started within a Command Prompt window. Output of trace sessions for Managed, Native, and UI Managed traces is output to each window.
The ViewRealTimeTracing.cmd batch file accepts a single, optional command line parameter (-ods). Supplying this parameter will additionally output trace data to a debugger.