Windows 10 Troubleshooting: Blue Screen Stop Code 0x0000021
Introduction
Ever met a situation where a Windows 10 Build 1803 machine not booting up and getting **BSOD (Blue Screen Of Death) **with Stop code 0x0000021?
This issue was noticed after machines were patched , however it may have been reboot of the machine that made it visible.
Troubleshooting
Disable Driver Signature enforcement check
To get the machine up we had to Disable Driver Signature Enforcement check -
Startup Options - F8 -> Disable Driver Signature Enforcement.
Run AV or Microsoft Safety scanner
First thing which you need to do is to run AV scan or you can use Microsoft safety scanner to make sure there is no malicious stuff going on, i would recommend to have your Security SME's to verify things
/en-us/windows/security/threat-protection/intelligence/safety-scanner-download** **
After machine was up we ran File Signature Verification (SigVerif) tool to see what files are not digitally signed. This tool is built into Windows and can be launched by using SigVerif command on command prompt or try SigVerif in search.
Verify files
After tool will finish scanning you will be able to see the files which are not digitally signed and here we are more concerned about kernel level drivers. In our case we had few in the list , however the one which stood out was HIDParse.sys. You can also analyze SIGVERIF.TXT log file for additional details.
Event ID 3004
Also you can check Event ID 3004 in CodeIntegrity Event Logs
**Event Viewer >> Application and Services Logs >>Microsoft >> Windows >> CodeIntegrity >> Operational **
Event ID - 3004
Windows is unable to verify the image integrity of the file \Device\HarddiskVolume1\Windows\System32\drivers\hidparse.sys because file hash could not be found on the system. A recent hardware or software change might have installed a file that is signed incorrectly or damaged, or that might be malicious software from an unknown source.
Memory dump analysis
We analyzed Memory Dump - MEMORY.DMP located at C:\Windows and also in C:\Windows\Minidump and both showed same stuff which we see in Event Logs :-
***************************************************************
* *
* Bugcheck Analysis *
* *
***************************************************************
WINLOGON_FATAL_ERROR (c000021a)
The Winlogon process terminated unexpectedly.
Arguments:
Arg1: ffff928d5f009550, String that identifies the problem.
Arg2: ffffffffc0000428, Error Code.
Arg3: 0000000000000000
Arg4: 000001ae80010000
STACK_TEXT:
fffff501`d1ed74a8 fffff803`3d51af84 : 00000000`0000004c 00000000`c000021a fffff501`d202d2f0 ffffba84`f48e2150 : nt!KeBugCheckEx
fffff501`d1ed74b0 fffff803`3d513138 : fffff501`d1ed75d0 fffff501`d1ed7570 ffffffff`8000066c fffff501`d1ed75d0 : nt!PopGracefulShutdown+0x294
fffff501`d1ed74f0 fffff803`3d50b798 : 00000000`00000601 fffff803`00000006 00000000`00000004 00000000`0002001f : nt!PopTransitionSystemPowerStateEx+0xcc58
fffff501`d1ed75b0 fffff803`3d24f143 : eea0db56`71e17019 fffff803`3d113731 00000000`00000010 00000000`00000082 : nt!NtSetSystemPowerState+0x4c
fffff501`d1ed7780 fffff803`3d2424c0 : fffff803`3d67ebf2 00000000`c0000004 ffffba84`f02f9a00 ffffba84`f2796180 : nt!KiSystemServiceCopyEnd+0x13
fffff501`d1ed7918 fffff803`3d67ebf2 : 00000000`c0000004 ffffba84`f02f9a00 ffffba84`f2796180 ffffba84`f2796140 : nt!KiServiceLinkage
fffff501`d1ed7920 fffff803`3d67e849 : 00000000`00000000 ffffba84`f02f9af0 ffffba84`f2796040 00000000`00000000 : nt!PopIssueActionRequest+0x292
fffff501`d1ed79e0 fffff803`3d1ce79b : 00000000`00000001 00000000`00000002 ffffba84`f02f9a00 00000000`00000000 : nt!PopPolicyWorkerAction+0x69
fffff501`d1ed7a50 fffff803`3d146155 : ffffba84`f2796040 fffff803`3d1ce720 ffffba84`f02f9af0 ffffba84`f02f9af0 : nt!PopPolicyWorkerThread+0x7b
fffff501`d1ed7a80 fffff803`3d0b8e27 : ffffba84`f2796040 00000000`00000080 ffffba84`f02bd040 ffffba84`f2796040 : nt!ExpWorkerThread+0xf5
fffff501`d1ed7b10 fffff803`3d245f66 : fffff803`37a5a180 ffffba84`f2796040 fffff803`3d0b8de0 00440031`00260034 : nt!PspSystemThreadStartup+0x47
fffff501`d1ed7b60 00000000`00000000 : fffff501`d1ed8000 fffff501`d1ed1000 00000000`00000000 00000000`00000000 : nt!KiStartSystemThread+0x16
ERROR_CODE: (NTSTATUS) 0xc000021a - {Fatal System Error} The %hs system process terminated unexpectedly with a status of 0x%08x (0x%08x 0x%08x). The system has been shut down.
EXCEPTION_CODE: (NTSTATUS) 0xc000021a - {Fatal System Error} The %hs system process terminated unexpectedly with a status of 0x%08x (0x%08x 0x%08x). The system has been shut down.
EXCEPTION_CODE_STR: c000021a
EXCEPTION_PARAMETER1: ffff928d5f009550
EXCEPTION_PARAMETER2: ffffffffc0000428
EXCEPTION_PARAMETER3: 0000000000000000
EXCEPTION_PARAMETER4: 1ae80010000
0: kd> !error ffffffffc0000428
Error code: (NTSTATUS) 0xc0000428 (3221226536) - Windows cannot verify the digital signature for this file. A recent hardware or software change might have installed a file that is signed incorrectly or damaged, or that might be malicious software from an unknown source.
Dump is showing us that machine is not booting up because a file doesn't have a digital signature - so we can say this the the result of the damage which has already been done but who did it still unknown.
So our focus went on HIDParse.sys file and we ran SysInternals - SigCheck64 tool and got following results :
c:\windows\system32\drivers\hidparse.sys:
Verified: Unsigned
Link date: 3:55 AM 11/24/2004
Publisher: n/a
Company: Microsoft Corporation
Description: Hid Parsing Library
Product: Microsoft« Windows« Operating System
Prod version: 10.0.17134.590
File version: 10.0.17134.590 (WinBuild.160101.0800)
MachineType: 64-bit
SigCheck
So here we can clearly see that HIDParse.sys doesn’t have its associated calalog file inside CatRoot Folder. In order to fix we temporarily copied HIDParse.sys from working machine and voila machine came up successfully.
Now we again ran SigCheck and got below results were we can clearly see file is signed with appropriate catalog (.cat) file -
Sigcheck v2.72 - File version and signature viewer
Copyright (C) 2004-2019 Mark Russinovich
Sysinternals - www.sysinternals.com
c:\windows\system32\drivers\hidparse.sys:
Verified: Signed
Link date: 6:02 PM 12/2/2010
Signing date: 12:35 PM 8/8/2019
Catalog: C:\Windows\system32\CatRoot\F750E6C3-38EE-11D1-85E5-00C04FC295EE}\Package_1531_for_KB4512501~31bf3856ad364e35~amd64~~10.0.1.8.cat
HIDparse.sys doesn't seem corrupt. There's an older version and a newer version. It seems that the catalog file for the older hidparse.sys got deleted but the file was not replaced. On reboot, this caused a mismatch when doing a catalog verification.
By replacing hidparse.sys with a newer version, the mismatch is resolved and the catalog file for the newer hidparse.sys is present on the system.
As impacted machine was having older version of HIDParse.sys and new catalog so easy and better option is to have associated catalog file added instead of replacing HIDParse.sys file. You can quickly check on machine which has signed version of older HIDParse.sys.
You can also enable verbose log for CodeIntegrity in Event Viewer.
Eg.
C:\Temp>sigcheck64.exe -i c:\windows\system32\drivers\hidparse.sys
Sigcheck v2.72 - File version and signature viewer
Copyright (C) 2004-2019 Mark Russinovich
Sysinternals - www.sysinternals.com
c:\windows\system32\drivers\hidparse.sys:
Verified: Signed
Link date: 3:55 AM 11/24/2004
Signing date: 5:46 PM 2/7/2019
Catalog: C:\Windows\system32\CatRoot\F750E6C3-38EE-11D1-85E5-00C04FC295EE}\Package_1440_for_KB4487017~31bf3856ad364e35~amd64~~10.0.1.7.cat
To test we copied .cat file from Working machine to non-working machine and rebooted and machine came up fine.
Catalog File Location : C:\Windows\system32\CatRoot\F750E6C3-38EE-11D1-85E5-00C04FC295EE}
Workaround
So workaround is to have missing catalog file added and its associated manifest file - MUM file. Below link from Microsoft has details around it:
https://support.microsoft.com/en-us/help/2700601/how-to-fix-errors-found-in-the-checksur-log
While looking for root cause why this issue happened we got a lead in log file, however it does not point out a direct root cause but we noticed with the impacted machines the log file “%SystemRoot%\inf\setupapi.dev.log” showing the following which might lead to possible cause:
inf: {Unconfigure Driver: Microsoft USB Natural Keyboard}
inf: Section Name = HID_Inst.NT
inf: {Add Service: HidUsb}
inf: Service 'HidUsb' still in use by 1 source.
inf: {Add Service: exit(0x00000000)}
inf: File 'C:\windows\System32\drivers\hidusb.sys' still in use by 4 sources.
inf: File 'C:\windows\System32\drivers\hidclass.sys' still in use by 4 sources.
inf: File 'C:\windows\System32\drivers\hidparse.sys' still in use by 4 sources.
inf: {Unconfigure Driver: exit(0x00000000)}
Who is holding HIDParse.sys? Work is in progress ...
Meanwhile Must Recommendations in situations where you see issues with catalog files :
- Run security scanner to make sure system is clean.
- Make sure Anti-Virus exclusions are set as per below MS Article which has information about scanning recommendations for Enterprise computers that are running currently supported versions of Windows:
- Anti-Virus Deep scanning and Windows security patching should not take place at same time.
- If there is any other security solution which has tendency of holding file for any sort of inspection , make sure above mentioned recommendations are followed appropriately.
- All EndPoint solutions are up-to-date and fully compatible with the version of Windows which you are running.