Freigeben über

Commonly reported issues while leveraging Windows Azure Diagnostic configurations and Best Practices

  • Artikel

In recent past we have noticed several customers reported issues related to Diagnostic configuration for Windows Azure PaaS Web/Worker Role. So we thought of blogging about some of the common symptoms, and resolution. We hope this blog should help the Windows Azure users to find the best practices for leveraging Windows Azure Diagnostic [WAD]. 

Summary

 

  1. OverallQuotaInMB is same as the Diagnostic store size.
  2. Keeping the OverallQutoInMB value greater than 4GB
    without configuring Diagnostic Store
  3. Keeping the child element value greater than
    OverallQuotaInMB
  4. ERROR_WINHTTP_AUTODETECTION_FAILED (12180)
  5. Best practices

 

Pre Requisite

 

If you are not familiar with Windows Azure Diagnostic configuration schema or some of the terminologies, I will recommend you to go through this  msdn article to understand the Windows azure Diagnostic Configuration Schema. Diagnostic configuration can either configured from the configuration file or programmatically inside the RoleEntryPoint.OnStart() Event.

 

If you are not familiar with Windows Debugging tool, please review this blog, and about managed debugging with sos.dll extension review this

 1.  OverallQuotaInMB is same as the Diganostic store size.

Symptom :       

  • The Diagnostic data were not transferring to the configured  specific storage account
  • CPU for the VM may be spiked to the  100% 
  • Size of the Windows Azure Diagnostic  Directory will show the  same size as the DiagnosticStore size  

 Root Cause : You should always make sure to keep the OverallQuotaInMB at least 500 MB less than configured size in  DiganosticStore (LocalStorage) value.  This detail is already mentioned in the below  msdn article :

 

Code Quick Start: Capturing diagnostics in your Windows Azure application

https://msdn.microsoft.com/en-us/library/windowsazure/hh180875.aspx

Extract from above article:

 “  The sum of the collective quotas used by the diagnostic logs must be less than or equal to the value of DiagnosticMonitorConfiguration.OverallQuotaInMB. Additionally, the DiagnosticMonitorConfiguration.OverallQuotaInMB
value must be 500 MB less than the value specified in the sizeInMB attribute of the <LocalStorage> element with name attribute set to DiagnosticStore (or less than or equal to 3584 MB if the sizeInMB attribute for the diagnostic
store is not specified in the service definition file). ”

 Resolution :

        Short term solution :  Delete the file in the directory.

        Long term solution  :  Modify the code to make sure OverallQuotaInMB, should be at least 500MB less than the value specified in the sizeInMB (DiagnosticStore) attribute and re-deploy the application.

 

 Please refer below mentioned configuration and code :

 ServiceDefinition.csdef

<?xml version="1.0" encoding="utf-8"?>

<ServiceDefinition name="Diagnostic_Sample" xmlns="https://schemas.microsoft.com/ServiceHosting/2008/10/ServiceDefinition">

  <WorkerRole name="WorkerRole1" vmsize="Small">

    <Imports>

      <Import moduleName="Diagnostics" />

    </Imports>

    <LocalResources>

      <LocalStorage name="DiagnosticStore" sizeInMB="8192" cleanOnRoleRecycle="false" />

    </LocalResources>

    <ConfigurationSettings>

      <Setting name="ScheduledTransferPeriod" />

      <Setting name="LogLevelFilter" />

    </ConfigurationSettings>

  </WorkerRole>

</ServiceDefinition>

 

Code

      public override bool OnStart()

        {

            <snip>

                DiagnosticMonitorConfiguration config = DiagnosticMonitor.GetDefaultInitialConfiguration();  //Diagnostic configuration start

                config.OverallQuotaInMB = 7692;

                config.Logs.BufferQuotaInMB = 1024;

                config.Directories.BufferQuotaInMB = 0; //use the rest of the storage here                
              

                config.WindowsEventLog.BufferQuotaInMB = 1024;

               config.PerformanceCounters.BufferQuotaInMB = 1024;

               config.DiagnosticInfrastructureLogs.BufferQuotaInMB = 1024;

            <snip>

            return base.OnStart();

        }

 (Note : This may be going to fix in next Windows Azure SDK release )

 

2.  Keeping the OverallQutoInMB value greater than 4GB without configuring Diagnostic Store :

Symptom :  

 

  • The Windows Azure Virtual Machine instances will be in busy state (Please refer this article for complete list of  Windows Azure instances status message displayed in the Management Portal) , and if you rdp into the Guest VM, and review the Windows Application Event Log, you will notice DiagnosticsAgent.exe is crashing. 
  • Attach the  Windows Debugging Tool to  DiagnosticsAgent.exe, and load the sos.dll extension, you can see below details :

   0:000> |

     .  0      id: 820       examine       name: E:\plugins\Diagnostics\DiagnosticsAgent.exe

   0:000> !threads

       ThreadCount:      2 

       UnstartedThread:0

       BackgroundThread: 1

       PendingThread:0

       DeadThread:0

       Hosted Runtime:no

                      
        PreEmptive Lock

       ID  OSID   ThreadOBJ     State GC       GC Alloc Context                  Domain           Count APT Exception

       0    1     824 00000000001ebfa0      a020 Enabled  0000000000000000:0000000000000000 00000000001d4770     0 MTA System.ArgumentOutOfRangeException (00000000011646e8) (nested exceptions)

        2    2   838 00000000001f12e0      b220 Enabled  0000000000000000:0000000000000000 00000000001d4770     0 MTA (Finalizer) 

                DiagnosticAgent.exe is throwing the ‘System.ArgumentOutOfRangeException’, if you dig more  into this exception, you will get something similar as below: 

0:000> !do 0000000001180138

Name: System.String

MethodTable: 000007fef8cb6980

EEClass:  000007fef883ed68

Size:  138(0x8a) bytes

File:   D:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll

String:      Specified argument was out of the range of valid values.

Fields:             
 MT                              Field         Offset                 Type    VT     Attr                  Value Name

000007fef8cbc868     4000103        8         System.Int32   1       instance           56 m_stringLength

000007fef8cbb3a8    4000104         c          System.Char  1       instance           53 m_firstChar

000007fef8cb6980    4000105       10        System.String  0       shared              static Empty

 >> Domain:Value 00000000001d4770:0000000001041420 <<

 0:000> !do 0000000001164668

Name: System.String

MethodTable: 000007fef8cb6980

EEClass: 000007fef883ed68

Size: 122(0x7a) bytes

File:        D:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll

String:      The overall quota cannot be greater than 4096MB.

Fields:             
    MT                       Field         Offset       Type              VT     Attr            Value      Name

000007fef8cbc868  4000103        8         System.Int32  1      instance      48           m_stringLength

000007fef8cbb3a8 4000104        c          System.Char  1      instance      54           m_firstChar

000007fef8cb6980 4000105       10        System.String  0     shared        static       Empty

 >> Domain:Value 00000000001d4770:0000000001041420 <<

                                                                                                                                 

 Additionally,if you want to find out the configuration values you can dump the managed stack object                                                                                                 

0:000> !dso

OS Thread Id: 0x824 (0)

RSP/REG                       Object                               Name

00000000001AC618      00000000011646e8        System.ArgumentOutOfRangeException

00000000001AC6B8     00000000011646e8         System.ArgumentOutOfRangeException

00000000001AC748     00000000011646e8         System.ArgumentOutOfRangeException

00000000001AC810     0000000001187880        System.String    Error starting diagnostics:{0}

00000000001AC8A0    0000000001187880          System.String    Error starting diagnostics:{0}

00000000001AC8B0     0000000001187880         System.String    Error starting diagnostics:{0}

00000000001AC8B8    0000000001187980         System.Object[]    (System.Object[])

00000000001AC8C0   00000000011646e8         System.ArgumentOutOfRangeException

<snip>

00000000001AEAF8 0000000001187980   System.Object[]    (System.Object[])

00000000001AEB08 00000000011646e8  System.ArgumentOutOfRangeException

00000000001AEB18 00000000011600c8 Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitorConfiguration

00000000001AEB28 0000000001052538 Microsoft.WindowsAzure.Plugins.Diagnostics.DiagnosticsExeKernel

<snip>

<snip>

 0:000> !do 00000000011600c8

Name: Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitorConfiguration

MethodTable: 000007ff001a8bc0

EEClass: 000007ff001c0d60

Size: 72(0x48) bytes

File:E:\plugins\Diagnostics\Microsoft.WindowsAzure.Diagnostics.dll

Fields:

             
MT                             Field   Offset                 Type VT     Attr            Value Name

000007fef8cd9330  4000044       38      System.TimeSpan  1 instance 0000000001160100 <ConfigurationChangePollInterval>k__BackingField

000007fef8cbc868  4000045       30         System.Int32  1 instance            32768 <OverallQuotaInMB>k__BackingField  <<<<< In this scenario the OverAllQuotaInMB configured as 32 GB

000007ff001ad7b0 4000046        8    ...fferConfiguration  0 instance    0000000001160150 <Logs>k__BackingField

000007ff001ad7b0 4000047       10 ...fferConfiguration  0 instance     0000000001160110 <DiagnosticInfrastructureLogs>k__BackingField

000007ff001ad890 4000048       18 ...fferConfiguration  0 instance   0000000001160170 <PerformanceCounters>k__BackingField

000007ff001ad9a0 4000049       20 ...fferConfiguration  0 instance  0000000001160190 <WindowsEventLog>k__BackingField

000007ff001ad378 400004a       28...fferConfiguration  0 instance 0000000001160130 <Directories>k__BackingField

 Resolution : The default value for the Diagnostic store is 4 GB, and if you want to make the Diagnostic store value greater than 4 GB, you should first configure the required  Diagnostic Store value

References :

DiagnosticMonitorConfiguration.OverallQuotaInMB Property

https://msdn.microsoft.com/en-us/library/microsoft.windowsazure.diagnostics.diagnosticmonitorconfiguration.overallquotainmb.aspx

 Extract from above link

 “The OverallQuotaInMB property specifies the amount of local storage allocated for the combined total of all the data buffers’ BufferQuotaInMB properties.

  By default, the OverallQuotaInMB is set to 4GB. If you want to specify a smaller amount, you can set this property to your desired value. You cannot specify a larger value using this property; instead you must add a <LocalStorage> element for DiagnosticStore to your ServiceDefinition.csdef file and change the sizeInMB attribute accordingly. For example:

   ServiceDefinition.csdef

 <LocalResources>

     <LocalStorage name="DiagnosticStore" sizeInMB="8192" cleanOnRoleRecycle="false"/>

 </LocalResources>

Note that if you specify a larger size in your ServiceConfiguration.cscfg file, such as 8GB in the example above, you must also set the OverallQuotaInMB property to the same value or a smaller value. If you make no changes to the
value of the OverallQuotaInMB property, your data buffers will be limited to the original 4GB value."

3.        Keeping the child element value greater than OverallQuotaInMB

       Symptom :    

 

  • The Role recycling, and if you remote  into the VM, you will noticed that DiagnosticsAgent.exe is crashing, and If you attached the WinDbg to the DiagnosticsAgent.exe, you will find below mentioned details.
  • Attach the  WindowsDebugging Tool  to DiagnosticsAgent.exe, and   load the sos.dll extension, you will see below details

 0:006> !threads

ThreadCount:      6

UnstartedThread:  0

BackgroundThread: 4

PendingThread:    0

DeadThread:       0

Hosted Runtime:   no

                                          
PreEmptive Lock 

  ID   OSID     ThreadOBJ                          State    GC         GC Alloc Context                                        Domain                      Count    APT  Exception

   0    1           448 000000000055cd60   200a020 Enabled  0000000000000000:0000000000000000 000000001a68ab60     0          MTA

   2    2            d88 0000000000562f70      b220 Enabled  0000000000000000:0000000000000000   0000000000550460     0 MTA (Finalizer)

   4   3             30c 000000001a68d900      1220 Enabled  0000000000000000:0000000000000000  0000000000550460     0 Ukn

   6   4            c54 000000001bde74d0       b020 Enabled  0000000000000000:0000000000000000  000000001a68ab60     0 MTA System.ArgumentException (00000000021cc6f0)

   7    6          14d0 000000001bdeff60       220 Enabled  0000000000000000:0000000000000000    0000000000550460     0 Ukn 

   5   5            df4 000000001bdef850       220 Enabled  0000000000000000:0000000000000000    0000000000550460     0 Ukn

 0:006> !do 00000000021cc6f0

Name:        System.ArgumentException

MethodTable: 000007fef6e0c1e0

EEClass:     000007fef6987390

Size:        168(0xa8) bytes

File: D:\windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll

Fields: 

 MT                          Field  Offset                Type     VT     Attr            Value                        Name

000007fef6df6728  400004b  8        System.String  0      instance    00000000021f71f0  _className

000007fef6df7750  400004c 10 ...ection.MethodBase  0 instance 0000000000000000 _exceptionMethod

000007fef6df6728  400004d 18        System.String  0 instance 0000000000000000 _exceptionMethodString

000007fef6df6728  400004e 20        System.String  0 instance 00000000021cc648 _message

000007fef6e19bb8  400004f 28 ...tions.IDictionary  0 instance 0000000000000000 _data

000007fef6df6af0  4000050 30     System.Exception  0 instance 0000000000000000 _innerException

000007fef6df6728  4000051 38        System.String  0 instance 0000000000000000 _helpURL

000007fef6df5880  4000052 40        System.Object  0 instance 0000000000000000 _stackTrace

000007fef6df5880  4000053 48        System.Object  0 instance 0000000000000000 _watsonBuckets

000007fef6df6728  4000054 50        System.String  0 instance 0000000000000000 _stackTraceString

000007fef6df6728  4000055 58        System.String  0 instance 0000000000000000 _remoteStackTraceString

000007fef6dfc610  4000056 88         System.Int32  1 instance 0                                 _remoteStackIndex

000007fef6df5880  4000057 60        System.Object  0 instance 0000000000000000 _dynamicMethods

000007fef6dfc610  4000058 8c         System.Int32  1 instance -2147024809 _HResult

000007fef6df6728  4000059 68        System.String  0 instance 0000000000000000 _source

000007fef6e03308  400005a 78        System.IntPtr  1 instance  0 _xptrs

000007fef6dfc610  400005b  90         System.Int32  1 instance -532462766 _xcode

000007fef6df6238  400005c  80       System.UIntPtr  1 instance      7ff004974e8 _ipForWatsonBuckets

000007fef6e2b5f0  400005d  70 ...ializationManager  0 instance 00000000021cc798 _safeSerializationManager

000007fef6df6728  400024d  98        System.String  0 instance 00000000021c0db0 m_paramName

 0:006> !do 00000000021cc648

Name:        System.String

MethodTable: 000007fef6df6728

EEClass:     000007fef697ed68

Size:        166(0xa6) bytes

File: D:\windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll

String:      OverallQuotaInMB is 9984MB but sum of requested sub-quotas is 17408MB.

Fields: 

 MT                          Field       Offset                 Type       VT     Attr            Value   Name

000007fef6dfc610  4000103  8         System.Int32          1 instance           70        m_stringLength

000007fef6dfb150  4000104  c          System.Char         1 instance           4f         m_firstChar

000007fef6df6728  4000105  10        System.String       0 shared              static   Empty

>> Domain:Value  0000000000550460:0000000001ed1420000000001a68ab60:0000000001ed1420 <<

 0:006> !do
000000000207e300

Name: Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitorConfiguration

MethodTable: 000007ff00510980

EEClass:     000007ff00503bb0

Size:        72(0x48) bytes

File:   E:\approot\Microsoft.WindowsAzure.Diagnostics.dll

Fields:

MT                            Field       Offset       Type                      VT     Attr            Value                       Name

000007fef6e190d8  4000044  38            System.TimeSpan  1       instance 000000000207e338  <ConfigurationChangePollInterval>k__BackingField

000007fef6dfc610  4000045   30             System.Int32          1       instance   9984                       <OverallQuotaInMB>k__BackingField

000007ff00516100  4000046   8              ...fferConfiguration  0      instance   000000000207e348 <Logs>k__BackingField

000007ff00516100  4000047  10             ...fferConfiguration  0      instance 000000000207e368 <DiagnosticInfrastructureLogs>k__BackingField

000007ff005161e0  4000048  18             ...fferConfiguration  0      instance 000000000207e388 <PerformanceCounters>k__BackingField

000007ff005162f0  4000049   20            ...fferConfiguration  0       instance 000000000207e3d0 <WindowsEventLog>k__BackingField

000007ff00515cc8  400004a 28             ...fferConfiguration  0        instance 000000000207e420 <Directories>k__BackingField

Resolution : Make sure to keep the child elements values less than the value configured for  OverallQuotaInMB. Please refer this  link to understand the child element details.
In this schema child elements are

  • DiagnosticInfrastructureLogs
  • Logs
  • Directories
  • WindowsEventLog

Child element in the code should be look like this

config.OverallQuotaInMB = 7692;

config.Logs.BufferQuotaInMB = 1024;

config.Directories.BufferQuotaInMB = 0; //use the rest of the storage here

config.WindowsEventLog.BufferQuotaInMB = 1024;

config.PerformanceCounters.BufferQuotaInMB = 1024;

config.DiagnosticInfrastructureLogs.BufferQuotaInMB = 1024;

4.        ERROR_WINHTTP_AUTODETECTION_FAILED

(12180) :   If you noticed the error like ‘WinHttpGetProxyForUrl(https://xxxxxxx.blob.core.windows.net) failed ERROR_WINHTTP_AUTODETECTION_FAILED (12180)", please refer this blog from my colleagues

 

5.    Bestpractices

 

  • Make sure OverallQuotaInMB, should be at least 500MB less than the value specified in the sizeInMB attribute.
  • If you want to configure the size of WAD log grater 4 GB, make sure to configure Diagnostic Store value to appropriate value.
  • Make sure to keep a separate storage account for storing the WAD Logs.
  • Refer the attached code sample, and in this code sample, I have configured the below values : 
  • Diagnostic store value 8 GB (refer .csfg file) Local storage.
  • OverallQuotaInMB value 7.5 GB
  • Sub-Quota limit 7.5 GB

 

References :

 Enabling Diagnostics in Windows Azure

https://www.windowsazure.com/en-us/develop/net/common-tasks/diagnostics/

 
Troubleshooting Best Practices for Developing Windows Azure Applications 

https://msdn.microsoft.com/en-us/library/windowsazure/hh771389.aspx

Author : Pradeep Kumar .C (CIE) Azure Escalation Services, Microsoft

Comments

  • Anonymous
    June 13, 2013
    Hi, The documentation here (msdn.microsoft.com/.../microsoft.windowsazure.diagnostics.diagnosticdatabufferconfiguration.bufferquotainmb.aspx) suggests that if the following values are all set to zero, they'll keep writing to the disk until OverallQuotaInMB is reached. However, we receive an error when OverallQuotaInMB is "3000": "OverallQuotaInMB is 3000MB but sum of requested sub-quotas is 3072MB.". The sum of requested sub-quotas should be "whatever OverallQuotaInMB is" shouldn't it? Is there a bug here or can't you set all these values to zero?config.OverallQuotaInMB = 0;config.Logs.BufferQuotaInMB = 0;config.Directories.BufferQuotaInMB = 0;config.WindowsEventLog.BufferQuotaInMB = 0;config.PerformanceCounters.BufferQuotaInMB = 0;config.DiagnosticInfrastructureLogs.BufferQuotaInMB = 0;
  • Anonymous
    June 14, 2013
    Thanks Greg for going through my blog, and it looks like you were facing the similar issue, which mentioned under scenario #3. Ideally you can set the value to '0' for the above mentioned parameter, and it should be taken the default value for OverallQuotaInMB (3596 MB [i.e 4096-500] ). Also, If you're not setting any value for the LocalStorage, 'DiagnosticStore', we need to investigate this issue to determine whether it is a bug. So could you please open a ticket for investigating this issue ?Also, please make sure whether you can reproduce the same behavior by leveraging Windows Azure SDK 2.0. In fact we have fixed few bugs in the SDK 2.0 [www.windowsazure.com/.../downloads].
  • Anonymous
    April 26, 2015
    Hi Pradeep ,where to find out the SizeInMb attribute. currently in diagnostic config it having the limit of ocerallquoteinmb=4096..