Windows 2008/ Windows 2008 R2 CNO Recovery:
What to do when you are unable to start the Cluster network Name (CNO) or any other Network name (VCO) in the Cluster,
Much of the information here is based on real issues and information from others that also experienced this issue,
Cluster Service Account:
Before Windows Server 2008, the cluster required the use of a Cluster Service Account (CSA). This was a domain user under whose credentials the cluster service, as well as cluster resources, ran. The CSA presented some problems, the most obvious of which was requiring administrators to rotate this password every so often.
How to manually re-create the Cluster service account
https://support.microsoft.com/kb/269229/en-us
How to change the Cluster service account password
https://support.microsoft.com/kb/305813/en-us
In Windows Server 2008, we removed this requirement. To replace the CSA, we created the Cluster Name Object (CNO). This is a Network Name resource that acts as the identity of the Cluster. This CNO in turn owns all of the Virtual Computer Objects (VCO) in the cluster. The VCOs are the computer names to which clients connect. The cluster service and cluster resources, now impersonate the CNO or the proper VCO.
To give an example, suppose you created a cluster named "FSCLUSTER" and this cluster hosts two applications, named "FS01" and "FS02." Active Directory will contain three computer objects - FSCLUSTER (CNO), FS01 (VCO), and FS02 (VCO). FSCLUSTER be the owner of FS01 and FS02.
More information about Active Directory with Failover Clustering, check our TechNet guide on Configuring Accounts for Active Directory: https://technet.microsoft.com/en-us/library/cc731002.aspx.
Description of the failover cluster security model in Windows Server 2008
https://support.microsoft.com/kb/947049
Recovering a Deleted Cluster Name Object (CNO) in a Windows Server 2008 Failover Cluster
When we see the Computer Object for the Cluster Name (Cluster Name Object (CNO) being removed from Active Directory resulting in the Cluster Name no longer being able to function properly.
This being done, it appears to us that the implications are not fully understood and there is no quick recovery from this.
1:
The first step to preventing this from happening in your organization is to be sure there is a clear understanding of the cluster security model in Windows Server 2008. Rather than spend a whole lot of time and space here rehashing what is already publicly available, I refer you to the following:
KB 947049: Description of the Failover Cluster Security Model in Windows Server 2008.
Failover Cluster Step-by-Step Guide: Configuring Accounts in Active directory
After reviewing the materials, you should have an understanding of how security works in Windows Server 2008 Failover Clusters and an appreciation for the importance of not removing (or disabling) the Computer Objects created in Active Directory by the cluster.
By default, the Computer Objects created by the cluster are all placed in the Computers container. These can be relocated to another OU, or even pre-staged in an OU before the cluster is created.
If pre-staging, be sure to review the requirements in the Step-by-step Guide already mentioned. As an example (Figure 1), I created a Cluster OU and moved the cluster nodes and their associated objects into the OU.
Figure 1
Figure 2
This cluster is located in the Cluster OU shown in Figure 1. Using Regedit.exe, I open the cluster registry hive and inspect the properties for the cluster. I can see the name of the cluster and the resource GUID for the Cluster Name.
Figure 3
Expanding the Resource GUID corresponding to the Cluster Name, I inspect additional properties for the resource. Selecting the Parameters entry displays the ObjectGUID for the cluster Computer Object in Active directory (Figure 4).
Figure 4
In Figure 5, we see the attribute in Active directory (must enable Advanced Features before the Attribute Editor tab is visible). You can also use ADSIEdit to view the same information.
Figure 5
The Cluster Name Object (CNO) functions as the primary security context for the cluster. The CNO is responsible for creating any additional Computer Objects (Virtual Computer Objects (VCO)) associated with the cluster. These Computer Objects represent Network Name resources in a cluster. A Network Name resource is created as part of a Client Access Point (CAP). Each Computer Object created by a cluster CNO contains an Access Control Entry (ACE) for the CNO on the Access Control List (ACL) for the object. The CNO is also responsible for synchronizing the password for each VCO in the domain. The VCOs associated with a particular CNO can be determined either by manually inspecting the ACL for each VCO in AD, or the information can be obtained in the cluster registry.
Opening the cluster registry hive and inspecting the properties of the Cluster Name resource, we can see an entry called ObjectGUIDS. This is a listing for each Computer Object created by the CNO in Active directory. In Figure 6, I have four Computer Objects in Active Directory associated with this cluster.
Figure 6
One of them is a Computer Object (VCO) associated with the CAP representing a highly available Print Server (CONTOSO-PS1) in this cluster (Figure 7).
Figure 7
So, what happens if the Cluster Name Object is deleted from Active Directory? A few important things –
· The Cluster Name, if Online, will stay Online but will fail to come Online again if the resource is cycled (it will be placed in a Failed state). This will prevent being able to connect to the cluster remotely when trying to administer the cluster.
· The security context for the cluster is lost. This prevents the passwords for all associated VCOs from being synchronized within the domain. Also, any user, service or other process needing permission to access cluster objects will fail to be authenticated.
· No more CAPs can be created in the cluster.
Besides the items listed above, there are other indications of problems. The Cluster Name resource in the Cluster Core Resources group will be in a Failed state. Attempts to bring the resource Online will generate a pop-up error (Figure 8)
Figure 8
A FailoverClustering error (Event ID 1207) will be registered in the System Log (Figure 9).
Figure 9
The cluster log will report a failure to locate the CNO Computer Object in Active Directory (Figure 10)
Figure 10
It is, therefore, very important the CNOs Computer Object in the domain not be deleted.
How do we recover from this?
1:
The supported way(s) to recover an Active Directory object that has been accidentally, or intentionally, deleted are described in the following articles and will not be covered in detail here–
KB840001: How to restore deleted user accounts and their group memberships in Active Directory
TechNet Content - Recovering Active Directory Domain Services
2:
Additionally, there are 3rd party solutions that can be used to protect Active Directory objects and\or recover them if deleted. Finally, as a last ditch effort, and when there is no other alternative, there is a free utility called ADRestore (32-bit only) that can be used to recover the Computer Object associated with the CNO. Please review the following information before deciding to use this utility –
Microsoft Supportability Newsletter – Using ADRestore tool to restore deleted objects
Either of these methods can be used, but they may end up being time consuming, expensive or both.
Once the Computer Object has been recovered from Active Directory, the Repair Active Directory object action can be used to restore functionality in the cluster (Figure 11).
Figure 11
Note: The logged on user that will perform the Repair action must have rights to administer the cluster and must have the right to Reset Passwords in the domain.
I personally believe ‘an ounce of prevention is worth a pound of cure.’ To that end, my top recommendation is to implement the steps outlined in the section Preventing unwanted deletions in the TechNet Content already mentioned above. Beginning with Windows Server 2008, objects in Active Directory, such as the Computer Object shown here (Figure 12), can be protected from accidental deletion by simply checking a box – Protect object from accidental deletion.
Figure 12
With this ‘guard’ in place, when an object is selected for deletion, the first pop-up is presented (Figure 13)
Figure 13
If Yes is selected, the next error is presented to the user (Figure 14) thus preventing deletion.
Figure 14
If this isn’t enough, there is more help coming in Windows Server 2008 R2. Domain Services in Windows Server 2008 R2 will include an optional feature called Active Directory Recycle Bin. This feature is not enabled by default and must be added. Details about the feature can be found on TechNet
TechNet Content – Active Directory Recycle Bin Step-by-Step Guide
Hugo Ferreira