Unable to Add Hosts from Untrusted Domain to the SCVMM 2012 R2 Server
Here is another very interesting issue that I worked on a few days back.
Let me first give you some history about the environment. We had a Few Untrusted Hosts in the perimeter network added to VMM and we tried to update the Host Agent to UR 6 on those Hosts, as the VMM was upgraded to UR 6 as well. But when we tried to upgrade the Agent of those Hosts it gave the below error:
Error 10436: Virtual Machine Manager does not support updating or re-associating an agent on a host that is in a non-trusted domain or on a perimeter network.
Which was expected according to the Blog Post below:
But then the customer removed those Hosts from Cluster from VMM. And then he was trying to add them back. But now the Host machines were not even getting added back to the Cluster and they stayed in Pending status. Also, we exported the Host Certificates from the Hosts to VMM’s Trusted People Store as it was not happening automatically. And we also updated the VMM agent on the Hosts to UR 6 version manually as well.
And then we again tried to Add the Hosts back to the Cluster in VMM, but it again failed. So, we collected the VMM traces while trying to add the Hosts in the cluster and we saw the below error there:
SecureCredential savedCredential, String& certificate) *** Carmine error was: CannotRetrieveCertificateError (20909); HR: 0x80072746 *** Host1.Contoso.Com ** The underlying connection was closed: An unexpected error occurred
And the Error Code 0x80072746 translates as below:
err 0x80072746
# as an HRESULT: Severity: FAILURE (1), FACILITY_WIN32 (0x7), Code 0x2746
# for hex 0x2746 / decimal 10054
WSAECONNRESET winerror.h
# An existing connection was forcibly closed by the remote
# host.
# 1 matches found for "0x80072746"
As we can see above error is showing that the Remote Host is closing the connection. So to check further on why the Host is closing the connection, we collected a simultaneous Network Capture on the VMM server and the Host while trying to reproduce the issue. And here is what we saw in the Network Capture:
Network Capture on VMM:
Same Communication on the Host:
So as we can see in the above Network Captures , we are getting a SSL Client HELLO on the host from the VMM server, but the Host is not responding with a SERVER HELLO on top of that, which we should expect to see after that. And that’s what looks to be the root cause here.
And because we see the behaviour mentioned above, we had to collect the WinRM listener’s output from the Host and as expected it shows that it is NOT listening on HTTPS on Port 5986, it only had a listener for HTTP Port 5986 as shown below and that is why it doesn’t have the Certificate Bound to it as well:
Output for the command winrm e winrm/config/listener:
Listener
Address = *
Transport = HTTP
Port = 5986
Hostname
Enabled = true
URLPrefix = wsman
CertificateThumbprint <<<< It is Blank
ListeningOn = 127.0.0.1, 169.254.2.213, Other IP Addresses
We should have expected to see a WinRM Listener for HTTPS Protocol on Port 5986, bound to a valid certificate, which was not the case here.
So we followed the following action plan to delete the current listener and create a new WinRM listener for Port 5986, for the correct protocol HTTPS and bound to a valid certificate of the Host:
- WinRM Listener configuration was adjusted
o Deleted the listener
winrm delete winrm/config/listener
o Recreated the listener
winrm create winrm/config/Listener?Address=*+Transport=HTTPS @{Hostname="<serverfqdn<";CertificateThumbprint="<SCVMM Agent certificate thumbprint>"}
After doing this we could see the listener correctly on the Host. Then we re tried to add the Hosts back to the VMM server and it was successful this time.
It was a lot of work done on this Issue, that’s why I wanted to share it with you all, so that if some one else hits the same issue, he would not have to put all these efforts again in resolving it.
I hope it will be helpful.
Author:
Nitin Singh
Support Escalation Engineer
Microsoft Security and Manageability Division