Windows 10 Troubleshooting: Upgrade OS Infinite Loop - VPN Connection
Issue
Post OS Upgrade installation the machine goes on an indefinite loop (restart won’t fix), leaves user unable to login to the system.
During this Covid Situation Many users would connect using VPN and end up with a broken machines and may need to go Office for support help.
Source
the issue is due to the “%WINDIR%\CCM\SetupCompleteTemplate.cmd” not completing the execution post upgrade of OS and hence the other post upgrade steps not gone through and the system goes on a loop, until the process is killed manually or a system hard reboot.
Why it breaks?
The upgrade process essentially breaks the chain with the task sequence (design), so this script is called as a post upgrade action to re-hook the task sequence failed to do its action.
Root Cause
Beginning in SCCM 1710 CB, the command to re-hook the task sequence post upgrade in the SetupCompleteTemplate.cmd has changed from:
%SCCMClientPath%\TSMBootstrap.exe /env:Gina /configpath:%_SMSTSMDataPath% /bootcount:2
to:
%CCMClientPath%\TSMBootstrap.exe /env:Gina /configpath:%_SMSTSMDataPath% /bootcount:2 /reloadenv
The /reloadenv switch resolves issues where a file not found error could occasionally be triggered (as it was in our case).
When machine starts to run “Upgrade Operating System”, it use “%WINDIR%\CCM\SetupCompleteTemplate.cmd” as sample to create “%WINDIR%\SMSTSPostUpgrade\setupcomplete.cmd”, then this cmd file goes to “C:\Windows.~BT\Sources\Scripts\setupcomplete.cmd”. After OSDUpgradeWindows is done, it goes to “%WINDIR%\Setup\Scripts\setupcomplete.cmd” – Where it’s not completing the execution, as it seems a native execution few machines may face this issue.
The SCCM environment to be upgraded to Current CB at a minimum, but also by deploying the latest clients as well. If your environment is automatically set up to update the client, you’re good to go with this. If you’re not or slow progress (which was our case), you will need to package the latest client and deploy it to the same deploy your upgrade task sequence or limit to latest agent version. It will typically run every time prior to launching the upgrade sequence.
Failure to update the client while deploying the task sequence will result in the task sequence immediately exiting post upgrade and failing to run the TSMBootstrap.exe hook due to the older client not knowing how to parse the /reloadenv switch at times (Not always).
Utilities version as per the latest agent installed devices.
Post OS upgrade the client devices execute the TS upon the re-hook controlled by local Agent as the version is mismatching here it gets stuck on the post upgrade setupcomplete.cmd and goes on an indefinite loop.
TsManager.exe – Entry Point not found” is caused by a mismatch between the SCCM integration with SCCM and the SCCM Client LSUtilities.dll – which provides Location Services for the SCCM Client.
Microsoft have made the assumption that if you are using the SCCM CB your client also in same version (though backward compatibility applicable). If you are running a stand-alone SCCM LTI build process, you may encounter this problem. The issues occur because SCCM integration modifies SCCM to work with the task sequence and the SCCM client. The SCCM client’s SCCM hooks still fire, but cannot function due to SCCM not having been upgraded.
BackGround
Multiple Machines which are having older version of client are unable to run the 'C:\WINDOWS\CCM\OSDUpgradeOS.exe' and client is trying for Client Remediation post the OS Upgrade and fails to start the CCMEXEC service even after successful remediation, after several retries client will try to communicate to MP and fails to load the TSENV
Which results in client gets stuck on the post upgrade setupcomplete.cmd and goes on an indefinite loop, since its stuck at the same step user unable to use the machine at all from Home network until the client is brought back to Office LAN Network and the TSENV able to reload post the client communication to MPMachines which are having latest client version are not checked for the client remediation and able to load the TS environment even in standalone network.
Resolution & Workaround
SCCM server and the client Agent to Same version before the Upgrade OS TS triggers as this issue may occur in few machines. Even though ((%SCCMClientPath%\TSMBootstrap.exe /env:Gina /configpath:%_SMSTSMDataPath% /bootcount:2 /reloadenv
The /reloadenv switch resolves issues where a file not found error could occasionally be triggered (as it was in our case)) Switch is in place.
Please upgrade the client to latest version before the OS Upgrade using Task sequence.
One can modify the setupcompletetemplate.cmd file to address this issue (last resort action not recommended).