Partilhar via

Detecting Low Virtual Memory Conditions in Windows 2008 and R2

  • Artigo

On Windows 2008 servers with Exchange 2007/2010 installed, there are times when you may run out of virtual memory for various reasons. One could be a memory leak in some application or simply not configuring the paging file correctly.

Once you run out of virtual memory on any given server, various applications may start failing/crashing on the server due to the inability to obtain memory to complete a specific function that is being called. In some cases, this could lead to a possible blue screen of death (BSOD).

For server based systems, the new Reliability Infrastructure helps automatically diagnose various operating system components. Of that infrastructure, Resource Exhaustion Detection and Resolution (RADAR) helps notify you when you are resources are reaching critical levels. RADAR is part of the Diagnostic Policy service that is installed on each server.

When RADAR detects that memory has reached a critical state, a 2004 event will be logged to the system log. An example of one of these events is shown below. As you can see, it has various information that provides overall memory consumption for various system resources, the top processes for memory consumption, file version information and paged/nonpaged pool memory that includes the top tags that could attribute to the memory problem. The bolded parts are the area of interest.

Log Name: System
Source: Microsoft-Windows-Resource-Exhaustion-Detector
Event ID: 2004
Task Category: Resource Exhaustion Diagnosis Events
Level: Warning
Keywords: Events related to exhaustion of system commit limit (virtual memory).
User: SYSTEM
Description:
Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: store.exe (7580) consumed 11282399232 bytes, MSExchangeMailboxAssistants.exe (21200) consumed 590950400 bytes, and w3wp.exe (21092) consumed 562757632 bytes.
Event Xml:
<Event xmlns="https://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Microsoft-Windows-Resource-Exhaustion-Detector" Guid="{9988748e-c2e8-4054-85f6-0c3e1cad2470}" />
<EventID>2004</EventID>
<Version>0</Version>
<Level>3</Level>
<Task>3</Task>
<Opcode>33</Opcode>
<Keywords>0x8000000020000000</Keywords>
<TimeCreated SystemTime="2010-09-03T10:47:01.431311400Z" />
<EventRecordID>169289</EventRecordID>
<Correlation ActivityID="{AC93AF3C-02AE-433D-8C22-FA32493FAD8C}" />
<Execution ProcessID="1160" ThreadID="8312" />
<Channel>System</Channel>
<Computer>Exserver01.domain.com</Computer>
<Security UserID="S-1-5-18" />
</System>
<UserData>
<MemoryExhaustionInfo xmlns:auto-ns2="https://schemas.microsoft.com/win/2004/08/events" xmlns="https://www.microsoft.com/Windows/Resource/Exhaustion/Detector/Events">
<SystemInfo>
<SystemCommitLimit>21261021184</SystemCommitLimit>
<SystemCommitCharge>20993597440</SystemCommitCharge>
<ProcessCommitCharge>19448094720</ProcessCommitCharge>
<PagedPoolUsage>453672960</PagedPoolUsage>
<PhysicalMemorySize>17176764416</PhysicalMemorySize>
<PhysicalMemoryUsage>17025470464</PhysicalMemoryUsage>
<NonPagedPoolUsage>422363136</NonPagedPoolUsage>
<Processes>133</Processes>
</SystemInfo>
     <ProcessInfo>
<Process_1>
<Name>store.exe</Name>
<ID>7580</ID>
<CreationTime>2010-09-02T11:21:32.755807700Z</CreationTime>
<CommitCharge>11282399232</CommitCharge>
<HandleCount>5619</HandleCount>
<Version>14.1.218.10</Version>
<TypeInfo>1089</TypeInfo>
</Process_1>
<Process_2>
<Name>MSExchangeMailboxAssistants.exe</Name>
<ID>21200</ID>
<CreationTime>2010-08-28T06:50:53.878440200Z</CreationTime>
<CommitCharge>590950400</CommitCharge>
<HandleCount>2664</HandleCount>
<Version>14.1.218.10</Version>
<TypeInfo>1090</TypeInfo>
</Process_2>
<Process_3>
<Name>w3wp.exe</Name>
<ID>21092</ID>
<CreationTime>2010-08-31T08:25:12.245594900Z</CreationTime>
<CommitCharge>562757632</CommitCharge>
<HandleCount>2817</HandleCount>
<Version>7.0.6002.18005</Version>
<TypeInfo>67</TypeInfo>
</Process_3>
<Process_4>
<Name>powershell.exe</Name>
<ID>19692</ID>
<CreationTime>2010-09-03T09:12:48.188589800Z</CreationTime>
<CommitCharge>152682496</CommitCharge>
<HandleCount>629</HandleCount>
<Version>6.0.6002.18111</Version>
<TypeInfo>136</TypeInfo>
</Process_4>
<Process_5>
<Name>mmc.exe</Name>
<ID>18768</ID>
<CreationTime>2010-09-03T09:12:42.167067000Z</CreationTime>
<CommitCharge>107646976</CommitCharge>
<HandleCount>464</HandleCount>
<Version>6.0.6002.18005</Version>
<TypeInfo>144</TypeInfo>
</Process_5>
<Process_6>
<Name>explorer.exe</Name>
<ID>13396</ID>
<CreationTime>2010-09-03T09:12:24.929288000Z</CreationTime>
<CommitCharge>22032384</CommitCharge>
<HandleCount>451</HandleCount>
<Version>6.0.6002.18005</Version>
<TypeInfo>152</TypeInfo>
</Process_6>
</ProcessInfo>
<PagedPoolInfo>
<Tag_1>
<Name>MmSt</Name>
<PoolUsed>216638928</PoolUsed>
</Tag_1>
<Tag_2>
<Name>CM31</Name>
<PoolUsed>103596032</PoolUsed>
</Tag_2>
<Tag_3>
<Name>MmRe</Name>
<PoolUsed>15907504</PoolUsed>
</Tag_3>
</PagedPoolInfo>
<NonPagedPoolInfo>
<Tag_1>
<Name>SmMs</Name>
<PoolUsed>161243168</PoolUsed>
</Tag_1>
<Tag_2>
<Name>BCM0</Name>
<PoolUsed>40694064</PoolUsed>
</Tag_2>
<Tag_3>
<Name>Cont</Name>
<PoolUsed>35498720</PoolUsed>
</Tag_3>
</NonPagedPoolInfo>
<ExhaustionEventInfo>
<Time>2010-09-03T10:47:18.540433800Z</Time>
</ExhaustionEventInfo>
</MemoryExhaustionInfo>
</UserData>
</Event>

This helps you determine what resource was the possible offender without having to install any additional tools on the server to troubleshoot this. The best part is that you don’t have to wait for an additional event to occur as the information has already been collected and logged.

There is another place where events are logged which is under the Windows Resource Exhaustion Detector (Resource-Exhaustion-Detector) under Applications and Services Logs in the Event Viewer as shown below.

image

These events show much less information than the system event, but do show your commit limits and charges to the system too. Sample below.

Log Name: Microsoft-Windows-Resource-Exhaustion-Detector/Operational
Source: Microsoft-Windows-Resource-Exhaustion-Detector
Event ID: 1003
Task Category: Resource Exhaustion Detection Events
Level: Warning
Keywords: Events related to exhaustion of system commit limit (virtual memory).
User: SYSTEM
Computer: ExServer01.Domain.Com
Description:
The Windows Resource Exhaustion Detector received a notification that the computer is low on virtual memory.
Event Xml:
<Event xmlns="https://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Microsoft-Windows-Resource-Exhaustion-Detector" Guid="{9988748e-c2e8-4054-85f6-0c3e1cad2470}" />
<EventID>1003</EventID>
<Version>0</Version>
<Level>3</Level>
<Task>2</Task>
<Opcode>22</Opcode>
<Keywords>0x4000000020000000</Keywords>
<TimeCreated SystemTime="2010-09-03T10:52:01.431065200Z" />
<EventRecordID>180</EventRecordID>
<Correlation ActivityID="{0B95CAB5-E004-4C92-BF5D-3BFA39FDF7EE}" />
<Execution ProcessID="1160" ThreadID="8312" />
<Channel>Microsoft-Windows-Resource-Exhaustion-Detector/Operational</Channel>
<Computer>ExServer01.domain.com</Computer>
<Security UserID="S-1-5-18" />
</System>
<UserData>
<CommitLimitExhaustion xmlns:auto-ns2="https://schemas.microsoft.com/win/2004/08/events"xmlns="https://www.microsoft.com/Windows/Resource/Exhaustion/Detector/Events">
      <SystemCommitLimit>21261021184</SystemCommitLimit>
<SystemCommitCharge>21258543104</SystemCommitCharge>
</CommitLimitExhaustion>
</UserData>
</Event>

A couple of potential events that can be seen when memory resources are low are shown below.

  • MSExchangeRepl Service failing to read a log file for database copy due to an out of memory error condition.

    Log Name: Application
    Source: MSExchangeRepl
    Event ID: 2168
    Task Category: Service
    Level: Error
    Keywords: Classic
    User: N/A
    Computer: ExServer01.Domain.Com
    Description:
    Log file F:\Exchsrvr\DB\DB0001\LOG001\E00000A7A46.log' for database copy EXServer MBX Store 001\ExServer01' couldn't be read. Error: Out of Memory (-1011)
    Event Xml:
    <Event xmlns="https://schemas.microsoft.com/win/2004/08/events/event">
    <System>
    <Provider Name="MSExchangeRepl" />
    <EventID Qualifiers="49156">2168</EventID>
    <Level>2</Level>
    <Task>1</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2010-09-02T11:21:13.000000000Z" />
    <EventRecordID>3190563</EventRecordID>
    <Channel>Application</Channel>
    <Computer>Exserver01.domain.com</Computer>
    <Security />
    </System>
    <EventData>
    <Data>F:\Exchsrvr\DB\DB0001\LOG001\E00000A7A46.log</Data>
    <Data>EXServer MBX Store 001\ExServer01' </Data>
    <Data>Out of Memory (-1011)</Data>
    </EventData>
    </Event>

  • A Registry flush operation failing to write the SOFTWARE hive to disk

    Log Name: System
    Source: Microsoft-Windows-Kernel-General
    Event ID: 6
    Task Category: None
    Level: Error
    Keywords:
    User: SYSTEM
    Computer: ExServer01.domain.com
    Description:
    An I/O operation initiated by the Registry failed unrecoverably.The Registry could not flush hive (file): '\SystemRoot\System32\Config\SOFTWARE'.
    Event Xml:
    <Event xmlns="https://schemas.microsoft.com/win/2004/08/events/event">
    <System>
    <Provider Name="Microsoft-Windows-Kernel-General" Guid="{a68ca8b7-004f-d7b6-a698-07e2de0f1f5d}" />
    <EventID>6</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8000000000000000</Keywords>
    <TimeCreated SystemTime="2010-09-03T10:48:17.714333400Z" />
    <EventRecordID>169290</EventRecordID>
    <Correlation />
    <Execution ProcessID="4" ThreadID="92" />
    <Channel>System</Channel>
    <Computer>ExServer01.domain.com</Computer>
    <Security UserID="S-1-5-18" />
    </System>
    <EventData>
    <Data Name="FinalStatus">0xc000014d</Data>
    <Data Name="ExtraStringLength">36</Data>
    <Data Name="ExtraString">\SystemRoot\System32\Config\SOFTWARE</Data>
    </EventData>
    </Event>

Depending on the component used to instantiate a specific function will determine what component will log the event in the system log. Finding root cause for memory issues has become significantly easier with this new Reliability Infrastructure and I hope this blog helps show you some of the methods for troubleshooting these type issues.

Until next time!!!

Comments

  • Anonymous
    January 01, 2003
    You work a lot for this. Thanks from all people that consider it is a step forward.

  • Anonymous
    June 07, 2013
    The comment has been removed

  • Anonymous
    June 07, 2013
    Why this Error Occurs and what is the procedure to remove these type of errors Log Name:      System Source:        Microsoft-Windows-TerminalServices-Printers Date:          07-Jun-13 10:22:05 AM Event ID:      1111 Task Category: None Level:         Error Keywords:      Classic User:          N/A Computer:      fs03.circasys.com Description: Driver Send To Microsoft OneNote 2010 Driver required for printer Send To OneNote 2010 is unknown. Contact the administrator to install the driver before you log in again. Event Xml: <Event xmlns="schemas.microsoft.com/.../event">  <System>    <Provider Name="Microsoft-Windows-TerminalServices-Printers" Guid="{952773BF-C2B7-49BC-88F4-920744B82C43}" EventSourceName="UmrdpService" />    <EventID Qualifiers="0">1111</EventID>    <Version>0</Version>    <Level>2</Level>    <Task>0</Task>    <Opcode>0</Opcode>    <Keywords>0x80000000000000</Keywords>    <TimeCreated SystemTime="2013-06-07T04:52:05.000Z" />    <EventRecordID>165880</EventRecordID>    <Correlation />    <Execution ProcessID="0" ThreadID="0" />    <Channel>System</Channel>    <Computer>fs03.circasys.com</Computer>    <Security />  </System>  <EventData>    <Data>Send To Microsoft OneNote 2010 Driver</Data>    <Data>Send To OneNote 2010</Data>    <Binary>00000000BA0E0000</Binary>  </EventData> </Event>

  • Anonymous
    May 15, 2014
    really helpful! Thank You!

  • Anonymous
    May 28, 2015
    i have system problem how i can solve

  • Anonymous
    May 28, 2015
    i have system problem how i can solve

  • Anonymous
    May 28, 2015
    i have system problem how i can solve

  • Anonymous
    May 28, 2015
    i have system problem how i can solve