Share via

Centralized Logging Service - Lync 2013 (CLS)

  • Article

Overview:
   

  • CLSController: Runs on any machine with 2013 Admin tools. Sends commands to CLSAgents running on Lync server roles
  • CLSAgent: Runs as a service on Lync server roles. Listens for commands from a CLSController on ports 50001,50002,50003
  • **Centralized Logging Service Agent: **ClsAgent.exe is the service executable that communicates with the controller and receives the commands that the controller is issued by the administrator. The agent is run as a service on each Lync Server computer. When the agent receives a command, it executes the command, sends messages to the defined components for tracing, and writes the trace logs to disk. It also reads the trace logs for its computer and sends the trace data back to the controller when requested. The ClsAgent listens for commands on the following ports: TCP 50001, TCP 50002, and TCP 50003.
  • Centralized Logging Service Controller: ClsControllerLib.dll is the command execution engine for the Lync Server Management Shell and for ClsController.exe. CLSControllerLib.dll sends Start, Stop, Flush, and Search commands to the ClsAgent. When search commands are sent, the resulting logs are returned to the ClsControllerLib.dll and aggregated. The controller is responsible for sending commands to the agent, receiving the status of those commands and managing the search log file data as it is returned from all agents on any computer in the search scope, and aggregating the log data into a meaningful and ordered output set. The information in the following topics is focused on using the Lync Server Management Shell. ClsController.exe is limited to a subset of the features and functions that are available in the Lync Server Management Shell. Help for ClsController.exe is available at the command line by typing ClsController in the default directory C:\Program Files\Common Files\Microsoft Lync Server 2013\ClsAgent.

**
Configuration**

  • CLS Configuration can be defined on Global, Site, Pool or Computer level

The Centralized Logging Service is governed by policy defined in powershell CsClsConfiguration. It can be run at a scope that includes a single computer, a pool of computers, at a site scope (Lync topology Site, not AD Site), or at a global scope. Global is included by default and most people will find using the global configuration to be flexible enough. Remember if you define narrower configurations, the narrowest policy wins (i.e. computer beats pool beats site beats global)

Settings in CsClsConfiguration (default values listed)

  • EtlFileFolder: %TEMP%\Tracing – When logging is started, the CLS service on the local machine will begin logging to an ETL file in this folder – remember the CLS runs as NetworkService, so %TEMP% here means %WINDIR%\ServiceProfiles\NetworkService\AppData\Local\Temp
  • EtlFileRolloverSizeMB: 20 – As the ETL file grows larger than 20MB, a new ETL file is created and the previous ETL file is converted to a .CACHE file and saved per the settings below
  • EtlFileRolloverMinutes: 60 – If an ETL file has not grown larger than 20MB in 1 hour, it is rolled over
  • CacheFileLocalFolders: %TEMP%\Tracing – where the ETL file goes when it is converted to a .CACHE file
  • CacheFileNetworkFolder: <not defined> - A location for long-term archival of log files
  • CacheFileLocalRetentionPeriod (days): 14 – After 14 days, the cache files are copied off the local storage to the CacheFileNetworkFolder (if defined), then deleted off local storage
  • CacheFileLocaMaxDiskUsage (percentage): 80 – The CLS will cease storing data when the drive defined in CacheFileLocalFolders is >80% consumed
  • ComponentThrottleLimit: 5000 – Maximum number of traces per second that a particular component can create before the throttle limiter is triggered
  • ComponentThrottleSample: 3 – Number of times in 60 seconds that the throttle limit can be exceeded before triggering the limiter (i.e. more than 5000 messages a second for 3 seconds out of 60 seconds will trigger component throttling – This will cause the component to discard traces over the limit, protecting the server from excessive logging load but discarding some of the traces in the process)
  • SearchTerms, SecurityGroups, Regions and MinimumClsAgentServiceVersion are meant for use with o365 deployments
  • TmfFileSearchPath can be used with updated TMF files and will largely be used when working with a CTS engineer. The value of null lets Lync format the trace files with RTM TMF files.

Providers

  • Providers map to OCSLogger components (Sipstack, S4 etc.)
  • These define:
  1. Component
  2. Logging level (All, Fatal, Error, Warning, Info, Verbose)
  3. Flags (TF_Connection, TF_Security, TF_Diag, TF_Protocol, TF_Component, All)
  • CLS Providers are always contained and triggered from within a Scenario

Providers If you are familiar with OCSLogger, providers are the components that you choose to tell OCSLogger what the tracing engine should collect logs from. The providers are the same components, and in many cases have the same name as the components in OCSLogger. If you are not familiar with OCSLogger, providers are server-role specific components that the Centralized Logging Service can collect logs from. In the case of the Centralized Logging Service, the CLSAgent is the architectural part of the Centralized Logging Service that is doing the tracing of the components that you define in the providers configuration.

Logging levels OCSLogger provided the option to choose a number of levels of detail for the data collected. This feature is an integral part of the Centralized Logging Service and scenarios, and is defined by the Type parameter. You can choose from the following:

  • All Collects trace messages of type fatal, error, warning, and info to the log for the defined provider.
  • Fatal Collects only the trace messages that indicate a failure for the defined provider.
  • Error Collects only the trace messages that indicate an error for the defined provider, plus fatal messages.
  • Warning Collects only the trace messages that indicate a warning for the defined provider, plus fatal and error messages.
  • Info Collects only the trace messages that indicate an informational message for the defined provider, plus fatal, error, and warning messages.
  • Verbose Collects all trace messages of type fatal, error, warning and info for the defined provider.
  • Flags OCSLogger provided the option to choose flags for each provider that defined what type of information you could retrieve from the trace files. You can chose the following flags, based on the provider:
  • TF_Connection Provides connection-related log entries. These logs include information about connections established to and from a particular component. This may also include significant network-level information (that is, for components without the concept of a connection)
  • TF_Security Provides all events/log entries related to security. For example, for SipStack, these are security events such as domain validation failure, and client authentication/authorization failures
  • TF_Diag Provides diagnostics events that you can use to diagnose or troubleshoot the component. For example, for SipStack, these are certificate failures, or DNS warnings/errors
  • TF_Protocol  Provides protocol messages such as SIP and CCCP messages
  • TF_Component Enables logging on the components specified as part of the providers.
  • All  Sets all available flags available for the provider
  • CLS Providers are always contained and triggered from within a Scenario.

Scenarios

  • Scenarios group providers based on specific troubleshooting goals
  • Most problems can be easily troubleshot using default scenarios
  • Custom Scenarios are created by:
  1. Creating the custom Providers
  2. Creating a custom Scenario containing the new providers

Scenarios allow you to group all of the providers that address a specific problem condition and execute them as a batch. Many scenarios are provided for you under the Global configuration and it is suggested that you explore the Providers contained in the default scenarios before creating a custom Scenario.

If you find that a scenario needs to be modified to meet your troubleshooting and logging needs, the Lync Server 2013 Debug Tools provides you a Windows PowerShell module named ClsController.psm1 that contains a function named Edit-CsClsScenario. The purpose of the module is to edit the properties of the named scenario.

As an example of how to create a custom Scenario with custom providers, we will make one called SimpleSip that grabs basic information from Sipstack and S4

To create a SipStack Provider getting everything

$sipStackProvider = New-CsClsProvider -Name SipStack -Type WPP -Level All -Flags All

To create a S4 provider, also getting everything, just defined specifically

$s4Provider = New-CsClsProvider -Name S4 -Type WPP -Level Info -Flags "TF_COMPONENT,TF_PROTOCOL“

Then we add them to a new scenario

New-CsClsScenario -Identity "Global/SimpleSip" -Provider $s4Provider,$sipStackProvider

Start and Stop Logging:

  • Start-CsClsLogging
  • -Scenario
  • -Computers
  • -Pools
  • -Duration
  • Stop-CsClsLogging
  • -Scenario
  • -Computers
  • -Pools

To start CLSLogging, use the Start-CsClsLogging command

Specify the Scenario using the Identity – this flag is required. For instance if you want to start the AddressBook scenario from the Global scope, use Global/Addressbook

With the –Computers and –Pools flags, specify which computers or Pools you want the scenario to run on. To specify more than one, use a comma separated list. Always use the FQDN of the computer or Pool. If nothing is specified for either option, the scenario will start on every computer in the scope of that scenario (i.e. if Global/Addressbook, the scenario will start on every computer in the Lync environment – if site:BNG-IN/SomeOtherScenario, it will start on every computer in the Lync site Redmond.

Duration specifies how long this scenario should run for – if omitted the default of 4 hours is used. Specify Duration using the notation “days.hours:minutes” – i.e. 6 days, 5 hours and 12 minutes would be –Duration 6.05:12. The alwaysOn scenario is an exception to this and runs until explicitly stopped as stated before.

  • To stop CLSLogging, use the Stop-CsClsLogging command

As before, Scenario is required, Computers and Pools are optional, but if omitted, will stop logging for the specified scenario for every computer in the scope for that scenario.

Searching

  • Search-CsClsLogging
  • -Computers, -Pools
  • -StartTime, -EndTime
  • -OutputFilePath
  • -Components
  • -LogLevel
  • -Phone, -IP, -URI, -ConferenceId
  • -MatchAll, -MatchAny
  • -SkipNetworkLogs

For those familiar with OcsLogger, think of the Search function as the evolution of the “Analyze Log Files” button. Like OcsLogger, Search lets you specify which components that you were tracing should be decoded and merged into a single clear text file. Unlike OcsLogger, Search also lets you filter those results by time, CallId, IP address, Log Level, Phone number, URI and SipContents. Additionally, it lets you aggregate trace outputs from multiple computers into the same merged clear text file.

All options on Search-CsClsLogging are optional, but filters are highly suggested. Default behavior if run without options is to run against every computer in the environment, from the time 25 min before the current time to 5 min after the current time, then display the output to the screen – not particularly useful.

  • Like start and stop, Computers and Pools limit the search scope to comma separated listes of computers or pools in the Lync environment
  • StartTime and EndTIme let you specify a time range to search in. If StartTime is specified but no EndTime, 30 min after the start time is used. Reverse is true is only EndTime is specified. If neither are specified, 25 min before now and 5 min after now are used
  • OutputFilePath – sends the output to the file specified rather than to the screen. If you specify a file name that already exists, the results will be appended to that file
  • Components – if you are running a scenario that collects a lot of components, but only want to see SipStack and S4, you can put these components into a comma separated list
  • LogLevel – you can filter on Log Level as well (Fatal,Error,Warning,Info,Verbose,Debug,All)
  • Results can be filtered by Phone number (E.164 format), IP, URI and ConferenceID as well. Wildcards are not allowed in these parameters.
  • MatchAll or MatchAny (default) are equivalent to “AND” and “OR” operations respectively
  • SkipNetworkLogs – search can not only look through logs on the computers specified, but search logs archived off to the network file share (as described under the Logging service configuration). Specifying this option causes the search to not query those logs

Snooper & ClsController

Snooper Tool

  • Once you have your search output file, what do you do with it? There are a number of ways that you can read the file. The output file is in a standard text format and you can use Notepad.exe or any other programs that will allow you to open and read a text file. For larger files and more complex issues, you could use a tool like Snooper.exe that is designed to read and parse the logging output from the Centralized Logging Service. Snooper is included with the Lync Server 2013 Debug Tools.
  • Like 2010 Snooper, 2013 snooper has two main tabs, one marked Trace, the other marked Messages. The trace tab will show every line of the log file indexed by component, level, flag, function, source, date/time, CorrelationID,Thread, Process, and CPU then the body of each trace is visible in the bottom half of the screen when that trace message is clicked on.
  • The Messages tab shows SIP traffic as well as Diagnostic information, with the time, Startline, From and To listed in a table on the left, then the full SIP message text selected from the table displayed on the right. When a message is clicked on in the left pane, all other messages that share the same Call-ID are highlighted as well. You can display all he messages with a particular Call-ID by right clicking and selecting “Find Related”
  • Prominent in both tabs is the search window. Text in traces or messages can be placed in here then the results are filtered out based on that text search. Strings separated by spaces are assumed OR operations (i.e. SERVICE sip:Alice@contoso.com would return any message with either Service OR sip:Alice.contoso.com in it). Enclosing your strings with double quotes (“) allows you to search terms with spaces
  • The Trace tab has a Filter option that can be used to filter and display only items based on trace headers
  • Reload File (F6) is often useful when troubleshooting a client log (Snooper’s great for reading .uccapilog files from the clients as well)
  • Notepad will open the log in notepad and auto-execute a find for the timecode of the message you currently have selected.
  • Log Selector allows you to open multiple log files from within Snooper and flip between them easily
  • The newest feature that adds value is the Flow Chart (call flow) diagram view. You select a message flow in the Message tab and click the Call Flow button. As you proceed through the messages, the call flow diagram updates with new data.
  • You can hover over the diagram view and get details about the messages and content of the flows and messages as well as the server elements. Click on any call flow arrow to go to the message in the Messages view.

ClsController - Tool

  • Import-Module C:\Program Files\Lync Server 2013\Debugging Tools\ClsController.psm1
  • Edit-CsClsScenario -ScenarioName <string of the scenario to edit> -ProviderName <string of the provider to remove> -Remove
  • Edit-CsClsScenario -ScenarioName <string of the scenario to edit> -ProviderName <string of the provider to add> -Level <string of type level> -Flags <string of type flags>

Included in the 2013 Debug tools, is a PowerShell module to assist you adding and removing providers from existing scenarios. The ClsController.psm1 module imports the Edit-CsClsScenario PowerShell commandlet into your shell. It allows you to easily add and remove providers from existing scenarios.

First you must import the module

  • Import-Module  C:\Program Files\Lync Server 2013\Debugging Tools\ClsController.psm1

To remove providers from an existing scenario

  • Edit-CsClsScenario -ScenarioName <string of the scenario to edit> -ProviderName <string of the provider to remove> -Remove

To add providers to an existing scenario

  • Edit-CsClsScenario -ScenarioName <string of the scenario to edit> -ProviderName <string of the provider to add> -Level <string of type level> -Flags <string of type flags>