Using Custom Performance Counters with Windows Azure Services

Windows makes available a wide variety of performance counters which of course are available to your Azure roles and can be accessed using the Azure Diagnostics APIs as I described in my recent MSDN article on Windows Azure Diagnostics.

However, it can be useful to create custom performance counters for things specific to your role workload.  For instance, you might count the number of images or orders processed, or total types of different data entities stored in blob storage, etc.

I wasn’t actually sure that I would have permissions to create custom performance counters and it turns out that right now you do not – all the code described below works in Development fabric (where you are running with rights not available to you in the Cloud).  Note as well that the symptoms when deploying to the Cloud with this code in place is that the role will continually be “Initializing” – it will never start.

Custom Performance Counters aren’t yet supported in the Azure Cloud fabric due to restricted permissions on Cloud-based Azure roles (see this thread for details) so this post describes how it will work when it is supported.

Create the Custom Performance Counters

1. You create your counters within a custom category.  You cannot add counters to a category, so you have to create a new performance counter category using PerformanceCounterCategory.Create in System.Diagnostics. The code below shows how to create a few different types of counters. Note that there are different types of counters: you can count occurrences of something (PerformanceCounterType.NumberOfItems32); you can count the rate of something occurring(PerformanceCounterType.RateOfCountsPerSecond32); or you can count the average time to perform an operation(PerformanceCounterType.AverageTimer32). This last one requires a "base counter" which provides the rate to calculate the average.

                // Create a new category of perf counters for monitoring performance in the worker role.
               try
               {
                   if (!PerformanceCounterCategory.Exists("WorkerThreadCategory"))
                   {
                       CounterCreationDataCollection counters = new CounterCreationDataCollection();
   
                       // 1. counter for counting totals: PerformanceCounterType.NumberOfItems32
                       CounterCreationData totalOps = new CounterCreationData();
                       totalOps.CounterName = "# worker thread operations executed";
                       totalOps.CounterHelp = "Total number of worker thread operations executed";
                       totalOps.CounterType = PerformanceCounterType.NumberOfItems32;
                       counters.Add(totalOps);
   
                       // 2. counter for counting operations per second:
                       //        PerformanceCounterType.RateOfCountsPerSecond32
                       CounterCreationData opsPerSecond = new CounterCreationData();
                       opsPerSecond.CounterName = "# worker thread operations per sec";
                       opsPerSecond.CounterHelp = "Number of operations executed per second";
                       opsPerSecond.CounterType = PerformanceCounterType.RateOfCountsPerSecond32;
                       counters.Add(opsPerSecond);
   
                       // 3. counter for counting average time per operation:
                       //                 PerformanceCounterType.AverageTimer32
                       CounterCreationData avgDuration = new CounterCreationData();
                       avgDuration.CounterName = "average time per worker thread operation";
                       avgDuration.CounterHelp = "Average duration per operation execution";
                       avgDuration.CounterType = PerformanceCounterType.AverageTimer32;
                       counters.Add(avgDuration);
   
                       // 4. base counter for counting average time
                       //         per operation: PerformanceCounterType.AverageBase
                       // NOTE: BASE counter MUST come after the counter for which it is the base!
                       CounterCreationData avgDurationBase = new CounterCreationData();
                       avgDurationBase.CounterName = "average time per worker thread operation base";
                       avgDurationBase.CounterHelp = "Average duration per operation execution base";
                       avgDurationBase.CounterType = PerformanceCounterType.AverageBase;
                       counters.Add(avgDurationBase);
   
   
                       // create new category with the counters above
                       PerformanceCounterCategory.Create("WorkerThreadCategory",
                               "Counters related to Azure Worker Thread", PerformanceCounterCategoryType.MultiInstance, counters);
                   }
                   catch (Exception exp)
                   {
                      Trace.TraceError("Exception creating performance counters " + exp.ToString());
                   }
   

2. When you create a performance counter category, you need to specify whether it is single-instance or multi-instance. MSDN helpfully explains that you should choose single-instance if you want a single instance of this category, and multi-instance if you want multiple instances. :) However, I found a blog entry from the WMI team that actually explains the difference - it is whether there is a single-instance of the counter on the machine (for Azure, virtual machine) or multiple instances; for example, anything per-process or per-thread is multi-instance since there is more than one on a single machine. As you can see, since I expect multiple worker thread role instances, I made my counters multi-instance.

3. After creating the counters, you need to access them in your code. I created private members of my worker thread role instance:

        public class WorkerRole : RoleEntryPoint
       {
   
     ...
   
           // Performance Counters
           PerformanceCounter  _TotalOperations = null;
           PerformanceCounter  _OperationsPerSecond = null;
           PerformanceCounter  _AverageDuration = null;
           PerformanceCounter  _AverageDurationBase = null;
     ...
   

4. Then in the code after I create the counter category if it doesn't exist, I create the members:

                try
               {
   
                   // create counters to work with
                   _TotalOperations = new PerformanceCounter();
                   _TotalOperations.CategoryName = "WorkerThreadCategory";
                   _TotalOperations.CounterName = "# worker thread operations executed";
                   _TotalOperations.MachineName = ".";
                   _TotalOperations.InstanceName = RoleEnvironment.CurrentRoleInstance.Id;
                   _TotalOperations.ReadOnly = false;
   
                   _OperationsPerSecond = new PerformanceCounter();
                   _OperationsPerSecond.CategoryName = "WorkerThreadCategory";
                   _OperationsPerSecond.CounterName = "# worker thread operations per sec";
                   _OperationsPerSecond.MachineName = ".";
                   _OperationsPerSecond.InstanceName = RoleEnvironment.CurrentRoleInstance.Id;
                   _OperationsPerSecond.ReadOnly = false;
   
                   _AverageDuration = new PerformanceCounter();
                   _AverageDuration.CategoryName = "WorkerThreadCategory";
                   _AverageDuration.CounterName = "average time per worker thread operation";
                   _AverageDuration.MachineName = ".";
                   _AverageDuration.InstanceName = RoleEnvironment.CurrentRoleInstance.Id;
                   _AverageDuration.ReadOnly = false;
   
                   _AverageDurationBase = new PerformanceCounter();
                   _AverageDurationBase.CategoryName = "WorkerThreadCategory";
                   _AverageDurationBase.CounterName = "average time per worker thread operation base";
                   _AverageDurationBase.MachineName = ".";
                   _AverageDurationBase.InstanceName = RoleEnvironment.CurrentRoleInstance.Id;
                   _AverageDurationBase.ReadOnly = false;
               }
               catch (Exception exp)
               {
                   Trace.TraceError("Exception creating performance counters " + exp.ToString());
               }
   

5. Note that I use "." for machine name, which just means current machine, and I use the CurrentRoleInstance.Id from the RoleEnvironment to distinguish the instance of each counter. If instead you wanted to aggregate these across the role rather than per-role instance, you could just use RoleEnvironment.CurrentRoleInstance.Name.

Using the Custom Performance Counters

Once you've created the performance counters, you now want to access them in your code - for instance, in the part of your worker role where you are actually doing work to track the time it takes to process a unit of work. As you know, Azure calls the Run method of your role to do the work, and this method should not return - it instead runs an infinite loop waiting for work to do and then doing it. Let's look at some simple code to use the performance counters defined above:

_TotalOperations	# worker thread operations executed
_OperationsPerSecond	# worker thread operations per sec
_AverageDuration	average time per worker thread operation
_AverageDurationBase	average time per worker thread operation base

Here's the code for Run that uses these; note that it uses the System.Diagnostics Stopwatch class which provides access to a higher-accuracy timer than simply calling DateTime.Ticks (see this blog post for more information)

         public override void Run()
        {
            while (true)
            {
                Stopwatch watch = new Stopwatch();
                // Test querying the perf counter.  Take a sample of the number of worker thread operations at this point.
                CounterSample operSample = _TotalOperations.NextSample();

                // Increment worker thread operations and operations / second.
                _TotalOperations.Increment();
                _OperationsPerSecond.Increment();

                // Start a stop watch on the worker thread work.
                watch.Start();

                // Do the work for this worker
                DoWork();

                // Capture the stop point.
                watch.Stop();

                // Figure out average duration based on the stop watch, then increment the base counter.
                _AverageDuration.IncrementBy(watch.ElapsedTicks);
                _AverageDurationBase.Increment();

                // Here's how you calculate the difference between a sample taken at the beginning and a sample at this point.
                // Note that since this is # of operations which is monotonically increasing, this is kind of a boring use of
                // a performance counter sample.
                Trace.WriteLine("Worker Thread operations performance counter: " + 
             CounterSample.Calculate(operSample, _TotalOperations.NextSample()).ToString());
            }
        }

This code also shows how to take two samples of a counter and calculate the difference - the counter I'm showing is rather uninteresting, but it illustrates the approach.

Monitoring the Performance Counters

OK, so you've added updating the performance counters in your code, but now you want to be able to see the values while your Azure process is running. The Azure Diagnostics Monitor collects this information in a local cache but by default does not persist it anywhere unless you ask it to. You can do this by adjusting the DiagnosticMonitorConfiguration with code like this (note this is from my role OnStart method):

             DiagnosticMonitorConfiguration dmc = DiagnosticMonitor.GetDefaultInitialConfiguration();
            TimeSpan tsOneMinute = TimeSpan.FromMinutes(1);
            TimeSpan tsTenMinutes = TimeSpan.FromMinutes(10);

            // Set up a performance counter for CPU time
            PerformanceCounterConfiguration pccCPU = new PerformanceCounterConfiguration();
            pccCPU.CounterSpecifier = @"\Processor(_Total)\% Processor Time";
            pccCPU.SampleRate = TimeSpan.FromSeconds(5);

            dmc.PerformanceCounters.DataSources.Add(pccCPU);

            // Add the custom counters to the transfer as well.
            PerformanceCounterConfiguration pccCustom1 = new PerformanceCounterConfiguration();
            pccCustom1.CounterSpecifier = @"\WorkerThreadCategory(*)\# worker thread operations executed";
            pccCustom1.SampleRate = TimeSpan.FromSeconds(20);
            dmc.PerformanceCounters.DataSources.Add(pccCustom1);

            PerformanceCounterConfiguration pccCustom2 = new PerformanceCounterConfiguration();
            pccCustom2.CounterSpecifier = @"\WorkerThreadCategory(*)\# worker thread operations per sec";
            pccCustom2.SampleRate = TimeSpan.FromSeconds(20);
            dmc.PerformanceCounters.DataSources.Add(pccCustom2);

            PerformanceCounterConfiguration pccCustom3 = new PerformanceCounterConfiguration();
            pccCustom3.CounterSpecifier = @"\WorkerThreadCategory(*)\average time per worker thread operation";
            pccCustom3.SampleRate = TimeSpan.FromSeconds(20);
            dmc.PerformanceCounters.DataSources.Add(pccCustom3);

            // Transfer perf counters every 10 minutes. (NOTE: EVERY ONE MINUTE FOR TESTING)
            dmc.PerformanceCounters.ScheduledTransferPeriod = /* tsTenMinutes */ tsOneMinute;

            // Start up the diagnostic manager with the given configuration.
            try
            {
                DiagnosticMonitor.Start("DiagnosticsConnectionString", dmc);
            }
            catch (Exception exp)
            {
                Trace.TraceError(""DiagnosticsManager.Start threw an exception " + exp.ToString());
            }

This code also adds a standard system performance counter for % CPU usage.

Note that you can either pass the DiagnosticMonitorConfiguration to DiagnosticMonitor.Start or you can change the configuration after the DiagnosticMonitor has been started:

            try
           {
        var diagManager = new DeploymentDiagnosticManager(
                      RoleEnvironment.GetConfigurationSettingValue("DiagnosticsConnectionString"),
                      RoleEnvironment.DeploymentId);
      var roleInstDiagMgr = diagManager.GetRoleInstanceDiagnosticManager(
                     RoleEnvironment.CurrentRoleInstance.Name,
                       RoleEnvironment.CurrentRoleInstance.Id);
        DiagnosticMonitorConfiguration dmc = diagManager.GetCurrentConfiguration();

     ... Make changes as needed to dmc, e.g. adding PerformanceCounters as above...

      roleInstDiagMgr.SetCurrentConfiguration(dmc);

Once you've transferred the performance counter data, they go into the Azure table storage for the storage account you've passed (i.e. the account specified by DiagnosticsConnectionString in ServiceConfiguration.cscfg). You can then either use a table storage browser to look at WADPerformanceCountersTable or you can use the very helpful Windows Azure Diagnostics Manager from Cerebrata Software (there is a free 30-day trial). This tool reads the raw data from the Azure table storage and allows you to download it, graph it, etc.

References

Thanks to the following which provided invaluable information along the way to figuring this out:

• Michael Groeger on CodeProject – An Introduction to Performance Counters
• Channel 9 – Monitoring Applications in Windows Azure
• Stack Overflow – Getting High Precision Timers in C#/.NET

Comments

• Anonymous
  July 20, 2010
  Hi Mike , If i am administrative privileges on my account , can i do it ? Thanks Chakka

• Anonymous
  July 20, 2010
  If i am running my webrole in admin mode , Can i achieve it ?