Azure HPC Cache
An Azure service that provides file caching for high-performance computing.
27 questions
HPC Pack 2019 Cluster manager cannot connect to newly installed cluster
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(57.599999999999994, 57.99999999999999%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EM%3C/text%3E%3C/svg%3E)
muellech
106
Reputation points
I have been upgrading on-premise HPC Pack 2016 to 2019 - basically a reinstall. The database is a completely new SQL Server instance and the scripts ran as expected. The installation went through w/o errors on three head nodes with built-in clustering. However, I cannot connect with the HPC Cluster Manager, presumably because the services are constantly crashing. Errors I see (on all nodes) in the application log include:
Application: HpcDiagnostics.exe Framework Version: v4.0.30319 Description: The process was terminated due to an unhandled exception. Exception Info: System.ComponentModel.Win32Exception at Microsoft.ComputeCluster.Management.Win32Helpers.HAUtils.SetGenericServiceRegistryCheckpoint(System.String, System.String) at Microsoft.Hpc.Diagnostics.Store.DiagnosticCrypto+<GetKeyAndSalt>d__23.MoveNext() at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task) at Microsoft.Hpc.Diagnostics.Store.DiagnosticCrypto+<InitDefault>d__18.MoveNext() at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task) at Microsoft.Hpc.Diagnostics.Store.DiagnosticsStore+<Init>d__27.MoveNext() at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task) at Microsoft.Hpc.Diagnostics.DiagnosticsSvc+<StartSvc>d__8.MoveNext() at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task) at Microsoft.Hpc.DiagnosticsWinService.DiagnosticsWinService+<<OnStart>b__2_1>d.MoveNext() at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task) at Microsoft.Hpc.HighAvailabilityModule.Algorithm.MembershipWithWitness+<>c__DisplayClass45_0.<RunAsync>b__0(System.Object) at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem() at System.Threading.ThreadPoolWorkQueue.Dispatch()
and
Application: HpcScheduler.exe Framework Version: v4.0.30319 Description: The process was terminated due to an unhandled exception. Exception Info: System.ComponentModel.Win32Exception at Microsoft.ComputeCluster.Management.Win32Helpers.HAUtils.SetGenericServiceRegistryCheckpoint(System.String, System.String) at Microsoft.Hpc.Scheduler.SchedulerCrypto+<InitKeyAndSalt>d__29.MoveNext() at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task) at Microsoft.Hpc.Scheduler.SchedulerCrypto+<InitDefault>d__24.MoveNext() at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task) at Microsoft.Hpc.Scheduler.Store.SchedulerStoreInternal..ctor(Boolean, Boolean, System.String, System.String, System.String, System.String, System.Func
2<System.String,System.String>, System.Func2<System.String,System.String>) at Microsoft.Hpc.Scheduler.Store.SchedulerStoreInternal..ctor(Boolean) at Microsoft.Hpc.Scheduler.SchedulerSvc+<StartSvc>d__19.MoveNext() at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task) at Microsoft.Hpc.Scheduler.SchedulerService+<<OnStart>b__5_1>d.MoveNext() at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task) at Microsoft.Hpc.HighAvailabilityModule.Algorithm.MembershipWithWitness+<>c__DisplayClass45_0.<RunAsync>b__0(System.Object) at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem() at System.Threading.ThreadPoolWorkQueue.Dispatch()
and the SCM reports that
The HPC Job Scheduler Service service terminated unexpectedly. It has done this 2300 time(s). The following corrective action will be taken in 30000 milliseconds: Restart the service.
The HPC Diagnostics Service service terminated unexpectedly. It has done this 2203 time(s). The following corrective action will be taken in 30000 milliseconds: Restart the service.
The HPC Session Service service terminated unexpectedly. It has done this 1843 time(s). The following corrective action will be taken in 30000 milliseconds: Restart the service.
The HPC SDM Store Service service terminated unexpectedly. It has done this 1811 time(s). The following corrective action will be taken in 30000 milliseconds: Restart the service.
The HPC Monitoring Server Service service terminated unexpectedly. It has done this 1825 time(s). The following corrective action will be taken in 30000 milliseconds: Restart the service.
The HPC Reporting Service service terminated unexpectedly. It has done this 1839 time(s). The following corrective action will be taken in 30000 milliseconds: Restart the service.
The HPC Management Service service terminated unexpectedly. It has done this 1837 time(s). The following corrective action will be taken in 30000 milliseconds: Restart the service.
I have already tried a complete rebuild of the cluster (including the DBs) to no avail. I think the issue might be related to TLS, because the call stack contains that crypto stuff (SchedulerCrypto+<InitKeyAndSalt>).
The certificate I use is the same on all three nodes from an AD-integrated CA and it is trusted by the nodes according to cert mgr.
I have TBH no idea how to proceed, because the exception information is not helpful at all.
{count} votes
-
Deleted
This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
-
muellech 106 Reputation points
2024-10-18T14:49:54.3766667+00:00 Another addition: Later in the management log, the certificate seems to be the right one:
10/18/2024 14:21:06.092 i HpcManagement.exe 6504 12184 [WcfProxy] Begin connect to endpointStr net.tcp://YYY:9893/SdmInternal, dnsIdentityName ZZZ
So my question is: where could it use the wrong certificate and is this a fatal issue?
-
muellech 106 Reputation points
2024-10-18T14:58:45.3433333+00:00 I now think that the following causes all subsequent errors:
10/18/2024 05:41:23.429 e HpcTrace 10116 13716 Current Application Domain UnhandledException event invoked: System.ComponentModel.Win32Exception (0x80004005): Failed to set registry checkpoint on service (error: 0).. at Microsoft.ComputeCluster.Management.Win32Helpers.HAUtils.SetGenericServiceRegistryCheckpoint(String serviceName, String RegPath).. at Microsoft.Hpc.Scheduler.SchedulerCrypto.<InitKeyAndSalt>d__29.MoveNext()..--- End of stack trace from previous location where exception was thrown ---.. at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw().. at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task).. at Microsoft.Hpc.Scheduler.SchedulerCrypto.<InitDefault>d__24.MoveNext()..--- End of stack trace from previous location where exception was thrown ---.. at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw().. at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task).. at Microsoft.Hpc.Scheduler.Store.SchedulerStoreInternal..ctor(Boolean restoreMode, Boolean schedulerOnAzure, String clusterName, String sqlString, String builtInAdmin, String builtInAdminPass, Func`2 azureUserPasswordDecryptor, Func`2 azureUserPasswordEncryptor).. at Microsoft.Hpc.Scheduler.Store.SchedulerStoreInternal..ctor(Boolean restoreMode).. at Microsoft.Hpc.Scheduler.SchedulerSvc.<StartSvc>d__19.MoveNext()..--- End of stack trace from previous location where exception was thrown ---.. at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw().. at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task).. at Microsoft.Hpc.Scheduler.SchedulerService.<<OnStart>b__5_1>d.MoveNext()..--- End of stack trace from previous l6�� where exception was thrown ---..��.�� ����[ .Exc xception J��� em.Runti �.�� HandleNonSuccessAndDebuggerN ion(� k).. a oft.Hpc.HighAvailabilityModule.Algorithm.MembershipWithWitness.<>c__DisplayClass45_0.<RunAsync>b__0(Object _).. at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx).. at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx).. at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem().. at System.Threading.ThreadPoolWorkQueue.Dispatch()The scheduler is in a restart loop due to this error, which causes the management service and all others to fail as well. Does anyone know what this "registry checkpoint" refers to?
Sign in to comment
Sign in to answer