次の方法で共有

WMI: High Memory Usage by WMI Service or Wmiprvse.exe

  • [アーティクル]

Scenario

Windows Management Instrumentation Service (Winmgmt) or WMI provider (wmiprvse.exe) is consuming high amounts of memory.

There are many reasons why WMI might experience high memory consumptions. This can occur in the WMI (Windows Management Instrumentation) service (winmgmt) or in the WMI provider hosting vehicle wmiprvse.exe. Both scenarios will be addressed below.

High memory usage may simply be due to load, as opposed to some type of leak. By default, the memory quota limit for instances of wmiprvse.exe on Windows XP and Windows Server 2003 is 128 MB, and 512 MB on newer Operating Systems (Vista and higher). Hit those limits and wmi functionality will become problematic if not come to a grinding halt.

There is another case where the quota limit problem could happen. There is limit for all wmiprvse’s cumulatively. If the total memory of all instances of wmiprvse’s (with one exception) together reaches 1GB, then all new memory allocations fail in all wmiprvse processes.

As a first effort you can try to bump up that quota limit to see if it gives enough room for wmiprvse to operate, but if you still reach the new quota limit, then you will want to proceed with the Scenarios below

How To Bump up WMI memory quota limit:

  • Go to Start--> Run and type wbemtest.exe
  • Click Connect

clip_image001 

 

  • In the namespace text box type "root" (without quotes).

Note: you aren’t connecting to CimV2 or any other namespaces. It’s ROOT

  • Click Connect
  • Click Enum Instances…

clip_image002

 

  • In the Class Info dialog box enter Superclass Name as "__ProviderHostQuotaConfiguration" (without quotes) and press OK. Note: the Superclass name includes a double underscore at the front.

clip_image003

 

  • In the Query Result window, double-click "__ProviderHostQuotaConfiguration=@"

clip_image004

 

  • In the Object Editor window, double-click whichever Property name you wish to modify the quota for. In this case that will be MemoryPerHost

clip_image005

 

  • In the Value dialog, type in 536870912 (512 MB) for XP and Windows 2003.  For Vista and higher, start with new value of 805306368 (768), then move to 1073741824 (1024) if still needing room. At the same time, if you are going to bump up the MemoryPerHostLimit, you should also then bump up the MemoryAllHost limit to 2147483648 (2048) provided you have the available ram to do so. If this does not resolve quota violation issue, then need to troubleshoot cause of high memory usage following below Scenario section
  • Click Save Property.
  • Click Save Object.
  • Close Wbemtest.
  • Restart the machine

Note: if you cannot restart the machine, then as a workaround, you can break Windows Management Instrumentation service into its own svchost process outline in step 2. a-c below.

If bumping the quota limits does not resolve the issue, then as workaround you can try to move suspected leaking providers into their own group (wmiprvse) to avoid the memory quota caused by other providers running in the group or kill the instance of wmiprvse exhibiting high memory until issue is resolved. This is outlined below in Scenarios 1a and 2a.

First thing we need to determine is if the memory consumption being caused by private data or heap data. We have to address the 2 types differently.

1. Download a Windows Sysinternals tool call VMMap from following url: https://technet.microsoft.com/en-us/sysinternals/dd535533.aspx

VMMap is a process virtual and physical memory analysis utility. It shows a breakdown of a process's committed virtual memory types as well as the amount of physical memory (working set) assigned by the operating system to those types.

This tool is used to attach to an individual process allowing a snapshot to be taken to see the memory map for that process.

2. Simply launch VMMap and from the process list it displays, pick the instance of wmiprvse showing the high working set memory usage.

3. If it is a svchost process that is exhibiting high memory, from a command run scqueryex winmgmt to identify the PID of the svchost process that is hosting WMI Service (winmgmt). From experience it will be the WMI service more times than not but not always as the service using majority of the memory; as such I would try to break it out first on its own and monitor to see if it is the one driving up high memory usage in the shared svchost process.

From this point, I will assume it is the WMI service as this article is only addressing WMI and no other services.

You will need to break the WMI service out into its own svchost process first if it hasn’t already been accomplished so you can run VMMap specifically against the WMI service to truly know if it is being consumed by Heap or Private Data. By default the WMI service runs in a shared svchost process with a community of other services. You can do so by the following directions. If it is caused by Heap, you will find later in the Scenario directions below that you will have to go back and uniquely name the svchost process for the service to run in to be able to enable User Stack Tracing against it.

a. Open command prompted with elevated privileges

b. Break wmi service out into its own svchost process by running following command: sc config winmgmt type= own

c. Restart wmi service with net stop winmgmt and net start winmgmt commands

d. Verify winmgmt service running in its own svchost process by runnning tasklist /svc

4. Once it displays the result, look under the size column for Private Data and Heap. This should tell you if the majority of the memory being consumed is Private Data or Heap.

5. Follow appropriate Scenario below based on if issue is with Windows Management Instrumentation service or wmiprvse and if caused by Heap or Private Data memory.

Scenario 1a: High Memory Consumption by Wmiprvse.exe caused by Heap memory

1. Download Windows Debugging Tools for Windows from following link: https://msdn.microsoft.com/en-us/windows/hh852365.aspx

You can probably just get the standalone version since we only need the debugging tool and not the whole WDK package.

2. Go to the directory where you installed the tool and you will find gflags.exe as one of the files, right click on it and select run as administrator

3. Click on “Image File” tab

4. Type in wmiprvse.exe

5. Hit the keyboard TAB key

6. Place check mark in “Create user mode stack trace database”

7. Click Ok

8. Download the latest version of the Windows Sysinternals tool Process Explorer. https://technet.microsoft.com/en-us/sysinternals/bb896653.aspx

9. Find the instance of wmiprvse.exe with high memory consumption and right click on it and bring up the properties sheet. Click on the WMI Providers tab and document the listed providers

Alternatively you could run following wmic command from command prompt to get a list of loaded providers and their associated PID you could then match up to PID with instance of wmiprvse with high memory usage:

wmic path msft_providers get Provider,HostProcessIdentifier /format:list

10. We now want to take that list of providers and configure them to run in their own instance of wmiprivse.exe. Open an instance of PowerShell with admin rights

11. Run the following commands: Replace ProviderName below with actual ProviderNamer found from step 9, retain the quote marks.

$prv = gcim -namespace root/standardcimv2 __win32provider -filter "name='ProviderName'"
<ENTER>
$prv.HostingModel = $Prv.HostingModel + ":OWN"
<ENTER>
set-ciminstance -inputobject $prv
<ENTER>

Repeat the above for each Provider found in step 9

12. Restarting of Wmi service will put the settings into effect without having to reboot machine

13. Now you will want to take a process dump of the instance of wmiprvse when memory usage is high. Download Windows Sysinternals tool called Procdump from URL: https://technet.microsoft.com/en-us/sysinternals/dd996900.aspx

14. Open a command prompt with elevated or administrative rights and change to the directory where you saved Procdump

15. Open Task Manager and add the PID column view then go locate the instance of wmiprvse.exe with high memory usage and note the PID, or use Process Explorer

16. Run the following command: procdump –ma -s 300 -n 3 <PID>

Note: Replace <PID> with actual PID you documented for instance of wmiprvse.exe exhibiting high memory usage

The above command will produce 3 dumps spaced at 5 minutes apart each in same directory you ran the procdump command from

17. Note also as workaround until issue gets resolved you could simply kill the instance of wmiprvse with high memory from task manager or from PowerShell can run following:

kill -f <pid of suspect wmiprvse>

At this point you will now need to open a Support Incident Case with Microsoft to get the data analyzed to determine cause of high memory usage

Reference this blog when you open the Support Incident Case with Microsoft as it will help the engineer understand what actions have been taken or followed and will help us track the effectiveness of the blog.

Scenario 1b: High Memory Consumption by WMI (Windows Management Instrumentation) service caused by Heap memory

The Windows Management Instrumentation service runs under the display name of winmgmt and is located in networking svchost.exe process shared along with several other services. You need to break the WMI service out into its own unique svchost process for data collection purposes.

NOTE: If you have already broken the WMI service out to run in its own general svchost process following directions at beginning of this article, then you can skip to step 6 to uniquely name the svchost process it is running in. We do this because we only want to enable User Stack Tracing against just the WMI service and not every svchost process.

  1. Open command prompted with elevated privileges
  2. Break wmi service out into its own svchost process by running following command: sc config winmgmt type= own
  3. Restart wmi service with net stop winmgmt and net start winmgmt commands
  4. Verify winmgmt service running in its own svchost process by runnning tasklist /svc
  5. Change command focus to system32 folder and run following command: copy svchost.exe wmisvchost.exe
  6. From start run type in regedit and navigate to HKLM\System_CurrentControlSet\Services\Winmgmt
  7. Modify existing ImagePath from %systemroot%\system32\svchost.exe -k netsvcs to %systemroot%\system32\wmisvchost.exe -k netsvcs

Note: It is important that you go back and reverse what you did in step 7 and modify path back to original after you are no longer needing the service to be broken out and uniquely named as failure to do this can prevent future WMI hotfixes from being installed.

Simply run following command then restart wmi service: sc config winmgmt type= share

8. Restart wmi service with net stop winmgmt and net start winmgmt commands again

9. Verify you now see wmisvchost.exe process running by running tasklist or looking in task manager at process list

10. For sake of brevity here, follow steps 4-14 from Scenario 1a with the exception of typing in wmisvchost.exe in place or wmiprvse.exe for process we are enabling “Create user mode stack trace database” against.

11. Here we want to use procdump to dump the wmisvchost.exe process along with every instance of wmiprvse.exe that is running. Reference steps 9-11 listed in Scenario 1a above.

First command will dump wmisvchost.exe 3 times spaced at 5 minute interval. Once it has completed, and then run second command to dump out each instance of wmiprvse.exe that is running.

Command for dumping wmisvchost.exe would be: procdump –ma -s 300 -n 3 wmisvchost.exe

Command for dumping each instance of wmiprvse just once: procdump –ma <PID>

Note: Replace <PID> with actual PID for each instance of wmiprvse.exe

At this point you will now need to open a Support Incident Case with Microsoft to get the process dumps analyzed to determine cause of high memory usage

Reference this blog when you open the Support Incident Case with Microsoft as it will help the engineer understand what actions have been taken or followed and will help us track the effectiveness of the blog.

Scenario 2a: High Memory Consumption by Wmiprvse.exe caused by Private Data memory

This scenario is a little bit different than being caused by Heap, as just dumps in of themselves normally do not tell the full story. In this type of case, we need to know what type of wmi activity is occurring and it becomes significantly harder and more involved to try and determine cause.

We will need to collect procdumps still, but also need to add in WMI activity logging.

1. From start run type in eventvwr.msc

2. On the View menu at the top select “show analytical and debug logs” so it displays check mark next to it

3. Expand Applications and Services\Microsoft\Windows\WMI-Activity

          Right click on items Debug and Trace and select “Enable” for each one

4. Download the latest version of the Windows Sysinternals tool Process Explorer https://technet.microsoft.com/en-us/sysinternals/bb896653.aspx.

5. Find the instance of wmiprvse.exe with high memory consumption and right click on it and bring up the properties sheet. Click on the WMI Providers tab and document the listed providers

Alternatively you could run following wmic command from command prompt to get a list of loaded providers and their associated PID you could then match up to PID with instance of wmiprvse with high

memory usage:

wmic path msft_providers get Provider,HostProcessIdentifier /format:list

6. We now want to take that list of providers and configure them to run in their own instance of wmiprivse.exe. Open an instance of PowerShell with admin rights

7. Run the following commands: Replace ProviderName below with actual ProviderNamer found from step 9, retain the quote marks.

$prv = gcim -namespace root/standardcimv2 __win32provider -filter "name='ProviderName'"
<ENTER>
$prv.HostingModel = $Prv.HostingModel + ":OWN"
<ENTER>
set-ciminstance -inputobject $prv
<ENTER>

Repeat the above for each Provider found in step 5

8. Restarting of Wmi service will put the settings into effect without having to reboot machine

9. When memory usage is high, use Procdump tool to dump out the wmiprvse.exe exhibiting high memory usage. https://technet.microsoft.com/en-us/sysinternals/dd996900.aspx

10. Open a command prompt with elevated or administrative rights and change to the directory where you saved Procdump

11. Open Task Manager and add the PID column view then go locate the instance of wmiprvse.exe with high memory usage and note the PID, or use Process Explorer

12. Run the following command: procdump –ma -s 300 -n 3 <PID>

Note: Replace <PID> with actual PID you documented for instance of wmiprvse.exe exhibiting high memory usage

The above command will produce 3 dumps space out 5 minutes apart each in same directory you ran the procdump command from

13. Collect a network trace for 2 minutes using Network Monitor. Latest version can be found at: https://www.microsoft.com/en-us/download/details.aspx?id=4865

14. Now save wmi-activity traces in Event Viewer, do so by right clicking on each and select “Save all events as” option

At this point you will now need to open a Support Incident Case with Microsoft for data analysis and most likely for more in depth troubleshooting and data collection methods

Reference this blog when you open the Support Incident Case with Microsoft as it will help the engineer understand what actions have been taken or followed and will help us track the effectiveness of the blog.

Scenario 2b: High Memory Consumption by WMI (Windows Management Instrumentation) service caused by Private Data memory

NOTE: If you have already broken the WMI service out to run in its own general svchost process following directions at beginning of this article, then you can skip to step 5

  1. Open command prompted with elevated privileges
  2. Break WMI service out into its own svchost process by running following command: sc config winmgmt type= own
  3. Restart WMI service with net stop winmgmt and net start winmgmt commands
  4. Verify winmgmt service running in its own svchost process by runnning tasklist /svc
  5. From start run type in eventvwr.msc
  6. On the View menu at the top select “show analytical and debug logs” so it displays check mark next to it
  7. Expand Applications and Services\Microsoft\Windows\WMI-Activity

          Right click on items Debug and Trace and select “Enable” for each one

8. Here we want to use procdump to dump the svchost process housing the WMI Service along with every instance of wmiprvse.exe that is running. Reference steps 9-10 listed in Scenario 1a above

a. From command prompt run tasklist /svc and locate the instance of svchost that winmgmt is running in and note the PID

b. First command will dump svchost.exe 3 times spaced at 5 minute interval. Once it has completed, and then run second command to dump out each instance of wmiprvse.exe that is running just once

Command for dumping svchost.exe housing winmgmt would be: procdump –ma -s 300 -n 3 <PID>

Note: Replace <PID> with actual PID for svchost process housing winmgmt

Command for dumping each instance of wmiprvse running would be: procdump –ma <PID>

Note: Replace <PID> with actual PID for each instance of wmiprvse.exe

 

  1. Collect a network trace for 2 minutes using Network Monitor. Latest version can be found at: https://www.microsoft.com/en-us/download/details.aspx?id=4865
  2. Now save wmi-activity traces in Event Viewer, do so by right clicking on each and select “Save all events as” option

At this point you will now need to open a Support Incident Case with Microsoft for data analysis and most likely for more in depth troubleshooting and data collection methods

Reference this blog when you open the Support Incident Case with Microsoft as it will help the engineer understand what actions have been taken or followed and will help us track the effectiveness of the blog.

Next up: WMI: How to troubleshoot High CPU Usage by WMI Components

Comments

  • Anonymous
    August 11, 2014
    Scenario Windows Management Instrumentation Service (Winmgmt) or WMI provider (wmiprvse.exe) is consuming