Hi, we are having trouble with repeated TDRs (Timeout Detection & Discovery) on multiple notebooks. They occur a couple of times a day. I've tried 1001 different things but nothing helped. So I've arrived here. My machine is a Lenovo notebook with an iGPU and an RTX 3000. The latter one is crashing (=TDR) randomly within no specific application. OS is Windows 11. I've tried new drivers and old drivers. Without DDU and with DDU. With and without a docking station With 2 external monitors (via display ports at the docking station) as well as only one external monitor directly connected to the notebook via HDMI. I've modified the registry to create a bug check on TDRs and analyze the memory.dmp (with WinDbg), but with no result. I've discovered PIX and looked into remote tracing, but it wants me to attach to a specific process, but I don't know to which one. The machine is not getting too hot and there is no game running, just Office applications. The bug can be reproduced e.g. by: Going to hibernate, and few seconds after waking up, the TDR occurs. But it also occurs randomly during normal usage or when just looking at the screen. Disabling the iGPU didn't help Most successful attempt so far: I've used WPR and activated "GPU activity" when starting. Unfortunately, the only type of recorded GPU activity was "Memory transfer". No rendering or any other kind of activity? It seems, MSFT doesn't provide a reference of the available trace providers anywhere (does it?). Luckily, the trace did include one "memory transfer" that took > 3 s, so it's obviously the one that lead to the TDR, restarting the graphics driver. But WHY did it take > 3 s?? I'm completely lost now. The same hardware problem on different machines? I don't think so. A driver bug that nobody but me seems to have for month now? Thanks in advance for any kind of hint. - Anything except reinstalling Windows.

Finding the actual reason for TDR

Armin Zingler 21

Hi,

we are having trouble with repeated TDRs (Timeout Detection & Discovery) on multiple notebooks. They occur a couple of times a day. I've tried 1001 different things but nothing helped. So I've arrived here.

My machine is a Lenovo notebook with an iGPU and an RTX 3000. The latter one is crashing (=TDR) randomly within no specific application. OS is Windows 11.

I've tried new drivers and old drivers.
Without DDU and with DDU.
With and without a docking station
With 2 external monitors (via display ports at the docking station) as well as only one external monitor directly connected to the notebook via HDMI.
I've modified the registry to create a bug check on TDRs and analyze the memory.dmp (with WinDbg), but with no result.
I've discovered PIX and looked into remote tracing, but it wants me to attach to a specific process, but I don't know to which one.
The machine is not getting too hot and there is no game running, just Office applications.
The bug can be reproduced e.g. by: Going to hibernate, and few seconds after waking up, the TDR occurs. But it also occurs randomly during normal usage or when just looking at the screen.
Disabling the iGPU didn't help
Most successful attempt so far:
I've used WPR and activated "GPU activity" when starting. Unfortunately, the only type of recorded GPU activity was "Memory transfer". No rendering or any other kind of activity? It seems, MSFT doesn't provide a reference of the available trace providers anywhere (does it?).
Luckily, the trace did include one "memory transfer" that took > 3 s, so it's obviously the one that lead to the TDR, restarting the graphics driver. But WHY did it take > 3 s??

I'm completely lost now.

The same hardware problem on different machines? I don't think so. A driver bug that nobody but me seems to have for month now?

Thanks in advance for any kind of hint. - Anything except reinstalling Windows.

Gary Nebbett 6,186 Reputation points

2024-11-01T10:21:14.96+00:00

Hello Armin,

Have you looked in %SystemRoot%\LiveKernelReports, %ProgramData%\Microsoft\Windows\WER and their subdirectories to see if the TDR event is captured in a live kernel dump?

Gary
Armin Zingler 21 Reputation points

2024-11-01T11:21:17.9566667+00:00

Hi Gary,

thanks, I've already analyzed the crush dump (see point #5 above). First I only had a minidump, then I've activated the full dump. It was a dead end for me. I had set TdrLevel = 1 as described in the documentation. In short, the stack says that it's initiating a bug check because of the timeout. But I've no clue why there is a timeout.
Gary Nebbett 6,186 Reputation points

2024-11-01T12:06:42.08+00:00

Hello Armin,

Was the bug check code 0x116 (VIDEO_TDR_FAILURE)? If so, was there a populated TDR_RECOVERY_CONTEXT?

When using WPR, are you sure that the trace period covered a TDR event? How did you identify the point in the trace when a TDR timeout occurred? Were the stack traces of events preceding the timeout helpful?

Gary
Armin Zingler 21 Reputation points

2024-11-04T17:22:21.1266667+00:00

Yes, there is a TDR_RECOVERY_CONTEXT. The documentation mentioned the command

dt dxgkrnl!_TDR_RECOVERY_CONTEXT ....

to inspect the content, but here it shows the message

Symbol dxgkrnl!_TDR_RECOVERY_CONTEXT not found

Symbols have been loaded, and I've verified that they are loaded for module dxgkrnl, too. However, the module doesn't seem to include this struct declaration. I've searched the web again. No solution. (Also tried this on another machine at home with Win 10 instead of Win 11: the same issue, i.e. the symbol is not found)

Another thing I've tried was installing the Windows Driver Kit. But again, there are no additional header files that contain the declaration of TDR_RECOVERY_CONTEXT. I've done a lot to find out where the struct is documented or declared, but I failed.

However, in WinDbg I've opened a memory window at the address of the TDR_RECOVERY_CONTEXT and pull out the information manually. The third field in the struct is "TimeoutReason". In my case the values is 6. But again, I do not know what that 6 means. In the example in the docs it is 9, which means "TdrEngineTimeoutPromotedToAdapterReset". Searching for this identifier on the web again, hoping to find the explanation of other possible values for the field, revealed nothing again. So I'm brachiating through this problem but nothing is tangible. :/

Regarding WPR: It was obvious in the graph itself when the TDR occured but also the point in time matched. I didn't know how to trace it back.
Armin Zingler 21 Reputation points

2024-11-05T16:05:11.0033333+00:00

More information:

I've analyzed the actions going on directly before the TDR. There were several processes activate, but I can only guess which one causes the problem. (Well, actually none of them should be able to cause it.)

Windows explorer was conspicuously active. Luckily it was performing graphics operations, so this increases the probability that it causes the problem. Almost all hits are inside win32u!NtGdiDdDDIDestroyHwQueue. Stack:

The frames inside LockController.dll are also interesting as they point to the actual task it's performing. But I didn't find good information about the purpose of the dll. Maybe regarding the lock screen. The whole graph:

"Tracing off" means hibernation. That's intentionally, because returning from hibernation is the way to reproduce the problem reliably. The red peak at the right is one single graphics action taking 3.39 s. This causes the TDR.
Armin Zingler 21 Reputation points

2024-11-05T16:07:03.7166667+00:00

EDIT: ignore

Gary Nebbett 6,186

Hello Armin,

Did you trim the (kernel) stack frames following win32u!NtGdiDdDDIDestroyHwQueue?

I would expect a Microsoft-Windows-DxgKrnl stack trace to end with a call to an ETW tracing routine. For example:

0xFFFFF8023502D77B=ntoskrnl!EtwWriteTransfer+0x2B
0xFFFFF8023FA02C04=dxgkrnl!McGenEventWrite_EtwWriteTransfer+0x48
0xFFFFF8023FA3FB92=dxgkrnl!McTemplateK0p_EtwWriteTransfer+0x42
0xFFFFF8023FC4C0B3=dxgkrnl!DXGCONTEXT::SubmitCommand+0xBC803
0xFFFFF8023FB904A7=dxgkrnl!DxgkSubmitCommandInternal+0x5B7
0xFFFFF8023FB8F877=dxgkrnl!DxgkSubmitCommand+0x67
0xFFFFF8023522ABA4=ntoskrnl!KiSystemServiceExitPico+0x468
# Kernel/User mode transition
0x7FFB9B8A5FB4=win32u!NtGdiDdDDISubmitCommand+0x14
0x7FFB93E873FC=d3d11!CallAndLogImpl<long (__cdecl*)(_D3DKMT_SUBMITCOMMAND const * __ptr64),_D3DKMT_SUBMITCOMMAND * __ptr+0x1C
0x7FFB93E7F4B6=d3d11!NDXGI::CDevice::SubmitCommandCB+0x166
0x7FFB86498269=igd10iumd64+0x138269
0x7FFB864988BE=igd10iumd64+0x1388BE
0x7FFB869FF0DC=igd10iumd64+0x69F0DC
0x7FFB869FEE53=igd10iumd64+0x69EE53
0x7FFB93E83C4C=d3d11!NDXGI::CDevice::PresentImpl+0xA2C
0x7FFB93E8316A=d3d11!NDXGI::CDevice::Present+0xFA
0x7FFB95451B7E=dxgi!CDXGISwapChain::DeviceInternalPresent+0x76
0x7FFB954537AF=dxgi!CFlipPresentToDWM<CDXGISwapChainWrapper>::FlipPresentCore+0x1EF
0x7FFB9545350A=dxgi!CDXGISwapChain::FlipPresentToDWM+0x8EA
0x7FFB954673F9=dxgi!CDXGISwapChain::PresentImplCore+0xD79
0x7FFB95466580=dxgi!CDXGISwapChain::PresentImpl+0xD0
0x7FFB9545407F=dxgi!CDXGISwapChain::[IDXGISwapChain4]::Present1+0xCF
0x7FFB49D890EF=Microsoft.Terminal.Control+0x790EF
0x7FFB49D881EE=Microsoft.Terminal.Control+0x781EE
0x7FFB49D5ADBD=Microsoft.Terminal.Control+0x4ADBD
0x7FFB49D5A9B5=Microsoft.Terminal.Control+0x4A9B5
0x7FFB9C54259D=kernel32!BaseThreadInitThunk+0x1D
0x7FFB9E0EAF38=ntdll!RtlUserThreadStart+0x28

Gary

Share via

Finding the actual reason for TDR

Your answer