Missing System Writer Case Explained
I worked on a case the other day where all I had was a procmon log and event logs to troubleshoot a problem where the System Writer did not appear in the VSSADMIN LIST WRITERS output. This might be review for the folks that know this component pretty well but I figured I would share how I did it for those that are not so familiar with the System Writer.
WHAT WE KNOW:
- System State Backups fail
- Running a VSS List Writers does not list the system writer
Looking at the event logs I found the error shown below. This error indicates there was a failure while “Writer Exposing its Metadata Context”. Each writer is responsible for providing a list of files, volumes, and other resources it is designed to protect. This list is called metadata and is formatted as XML. In the example we are working with the error is “Unexpected error calling routine XML document is too long”. While helpful, this message alone does not provide us with a clear reason why the XML document is too long.
Event Type: Error
Event Source: VSS
Event ID: 8193
Description: Volume Shadow Copy Service error: Unexpected error calling routine XML document is too long. hr = 0x80070018, The program issued a command but the command length is incorrect. . Operation: Writer Exposing its Metadata Context: Execution Context: Requestor Writer Instance ID: {636923A0-89C2-4823-ADEF-023A739B2515} Writer Class Id: {E8132975-6F93-4464-A53E-1050253AE220} Writer Name: System Writer
The second event that was logged was also not very helpful as it only indicates that the writer did have a failure. It looks like we are going to need to collect more data to figure this out.
Event Type: Error
Event Source: VSS
Event ID: 8193
Description: Volume Shadow Copy Service error: Unexpected error calling routine CreateVssExamineWriterMetadata. hr = 0x80042302, A Volume Shadow Copy Service component encountered an unexpected error. Check the Application event log for more information. . Operation: Writer Exposing its Metadata Context: Execution Context: Requestor Writer Instance ID: {636923A0-89C2-4823-ADEF-023A739B2515} Writer Class Id: {E8132975-6F93-4464-A53E-1050253AE220} Writer Name: System Writer
From the error above we learned that there was an issue with the metadata file for the System Writer. These errors are among some of the most common issues seen with this writer. There are some not so well documented limitations within the writer due to some hard set limits on path depth and the number of files in a given path. These limitations are frequently exposed by the C:\Windows\Microsoft.Net\ path. Often, this path is used by development software like Visual Studio as well as server applications like IIS. Below I have listed a few known issues that should help provide some scope when troubleshooting System Writer issues.
Known limitations and common points of failure:
- More than 1,000 folders in a folder causes writer to fail during OnIdentify
- More than 10,000 files in a folder causes writer to fail during OnIdentify (frequently C:\Windows\Microsoft.Net)
- Permissions issues (frequently in C:\Windows\WinSXS and C:\Windows\Microsoft.Net)
- Permissions issues with COM+ Event System Service
- This service needs to be running and needs to have Network Service with Service User Rights
What data can I capture to help me find where the issue is?
The best place to start is with a Process Monitor (Procmon) capture. To prepare for this capture you will need to download Process Monitor, open the Services MMC snap-in, as well as open an administrative command prompt which will be used in a later step of the process.
You should first stop the Cryptographic Services service using the Services MMC.
Once stopped you will want to open Procmon, note that by default Procmon will start capturing when opened. Now that you have Procmon open and capturing data you will start the cryptographic service. This will allow you to capture any errors during service initialization. Once the service is started you will use the command prompt opened earlier to run “vssadmin.exe list writers”. This will signal the writers on the system to capture their metadata, which is a common place we see failures with the System Writer. When the vssadmin command completes, stop the Procmon capture and save this data to disk.
Now that we have data how do we find the errors?
Open your newly created Procmon file. First, add a new filter for the path field that contains “VSS\Diag”.
We do this because this is the registry path that all writers will log to when entering and leaving major functions. Now that we have our filtered view we need to look for an entry from the System Writer. You can see the highlighted line below shows the “IDENTIFY” entry for the System Writer. From here we can ascertain the PID of the svchost.exe that the system writer is running in. We now want to include only this svchost. To accomplish this you can right click on the PID for the svchost.exe and select “Include ‘(PID)’”.
Now that we have found our System Writers svchost we will want to remove the filter for “VSS\Diag”; to do that you can return to the filter tool in Procmon and uncheck the box next to the entry.
We now have a complete view of what this service was doing at the time it started and completed the OnIdentify. Our next step is to locate the IDENTIFY (Leave) entry as this is often a great marker for where your next clue will be. While in most cases we can’t directly see the error the writer hit we can make some educated connections based on the common issues we spoke about above. If we take a look at the events that took place just before the IDENTIFY (Leave) we can see that we were working in the C:\Windows\Microsoft.NET\assembly\ directory. This is one of the paths that the System Writer is responsible for protecting. As mentioned above, there are some known limitations to the depth of paths and number of files in the “C:\Windows\Microsoft.NET” folder. This is a great example of that limitation as seen in our procmon capture. The example below shows the IDENTIFY (Leave) with the line before that being where the last work was taking place. Meaning this is what the writer was touching when it failed.
What does this tell us and what should we do next?
Given the known path limitations, we need to check out the number of files and folders in the C:\Windows\Microsoft.Net\ path and see where the bloat is. Some of these files can be safely removed, however only files located in the Temp locations (Temporary ASP.NET Files) are safe to delete.
Recently we released KB2807849 which addresses the issue shown above.
There are other possible causes of the event log errors mentioned above, such as issues with file permissions. For those problems follow the same steps as above and you are likely to see the IDENTIFY (Leave) just after file access error is displayed in your procmon log. For these failures you will need to investigate the permissions on the file we failed on. Likely the file is missing permissions for the writer’s service account Network Service or Local System. All that is needed here is to add back the missing permissions for your failed file.
While these issues can halt your nightly backups, it is often fairly easy to find the root cause. It just takes time and a little bit of experience with Process Monitor.
Good luck and successful backups!
Comments
Anonymous
August 28, 2013
Very useful, especially mentioning the VSSDiag, thanks!Anonymous
September 03, 2013
The comment has been removedAnonymous
July 17, 2014
very usefull and clear, so i have discovered that my server have 1242 files at path C:WindowsMicrosoft.NETassemblyGAC_MSIL so i asked how can i know what files could be safely removed? did not find any temp file on that path. Thanks a lot [Only files located in the Temp locations (Temporary ASP.NET Files) are safe to delete. You may find that http://support.microsoft.com/kb/2807849 applies to this problem.]Anonymous
December 08, 2014
Thanks to this technique I was able to identify problematic files (C:WindowsInfsetupapi.ev1-3) in my Windows 2008 x64. I've replaced them with working copies from another server. Actually I had the same issue as described here: www.cluberti.com/.../vss-system-writer-missing-no-cryptsvc-or-capi-errors-no-problem Big Thanks goes to You and clubertiAnonymous
December 12, 2014
The comment has been removedAnonymous
December 19, 2014
I've got this problem right now on a server with relatively little installed. It's Server 2012 R2 Standard with SQL Server 2008 R2 and not much else. The hotfix doesn't work for me - it says it is the wrong operating system. Is there another solution? Mike [The hotfix is already included in Windows Server 2012 R2 and is not needed on that platform.]