Disaster Recovery Procedures for Active Directory Certificate Services (ADCS)
Introduction:
When designing a public key infrastructure (PKI) for your organization, you must develop an effective disaster recovery plan to ensure that, in the event of failure of the computer hosting Certificate Services, you can recover in a timely manner with little effect on your organization.
Common Reasons that Make a Disaster Recovery Plan Necessary Include:
Failed services. If Certificate Services fails to start on the certification authority (CA) computer, no certificate can be issued and certificate revocation lists (CRLs) cannot be published. Your disaster plan for recovery should include performing either System State or manual CA backups and testing recovery (on a different system) on a regular basis.
Hardware failure. Disaster plan options for recovering after hardware failure include:
· Maintaining duplicate hardware (such as spare motherboards or spare computers);
· Implementing fault-tolerant RAID 1 or RAID 5 volumes to prevent CA failure due to a single disk failure.
Network infrastructure failure. Disaster recovery plans must account for network infrastructure failures. If an application implements CRL checking and network infrastructure failure prevents the application from accessing the most recent version of the CRL, an application will not validate the certificates presentedto the application. Your disaster recovery should include methods of diagnosing network infrastructure failures and developing methods of publishing CRL information that are redundant to protect against network failure.
Developing Required Documentation
One of the most important tasks during the design and deployment of a PKI is to ensure that your network and configuration documentation is updated continually. When you undergo disaster recovery, this documentation is the most important source of information regarding the previous Certificate Services configuration.
You should maintain the following documentation to ensure that you can apply all required configuration of Certificate Services successfully. The backup and recovery procedure for each of these items is explained later in this document.
All certificate template definitions. In the worst case, you might have to rebuild Active Directory, which requires the redefinition of all certificate templates. By documenting the individual settings for each certificate template on a tab-by-tab basis, you can easily re-create each certificate template.
All certificate templates published at the CA. You can create a custom script file that implements the certutil –SetCAtemplates +<TemplateName> to publish certificate templates and certutil –SetCAtemplates –<TemplateName> to remove certificate templates from the CA.
All permissions and user rights assignments. CA permissions define which users or groups hold the CA administrator and certificate manager Common Criteria roles, which groups or users can read the CA configuration, and which groups or users can request certificates from the CA. In addition, the Local Security Policy or domain-based Group Policy objects (GPOs) applied to the CA’s computer account defines the user rights assigned to the computer account, including the Common Criteria backup operators and auditor role holders.
All names used for the CA. Includes the CA’s logical name, the NetBIOS name of the computer hosting Certificate Services, and the domain or workgroup membership. The certificate information is based on the CA’s specific names and must be restored exactly.
All specific settings in the properties of the CA in the Certification Authority console. Be sure to identify which certificates are designated for key recovery, if implemented, as well as certificate manager restrictions.
Any post- or preinstallation script files used to configure the CA. For example, if you run a batch file consisting of certutil commands that define the CA’s registry settings, you should store a copy of the batch file for documentation and recovery purposes. Likewise, you should keep a copy of a batch file that publishes the CA’s CRL on an externally accessible Web server.
The CA data paths. When you restore the CA, the previous file locations for the CA database, CA log files, and CA configuration information must be maintained to match the restored registry values.
The CRL and Authority Information Access (AIA) publication points. Once the CA is restored, you must publish an updated CRL and, possibly, an updated CA certificate to the designated publication points. Ensure that no previous publication points are omitted.
The cryptographic service provider (CSP) used to protect the CA’s privatekey. The same CSP must be used to restore the previous key pair for theCA. The CSP might require additional software.
The key length of the CA’s certificate. If you are reinstalling the CA orrenewing the CA certificate, you should maintain the same key length as originallydeployed.
The logical disk-partitioning scheme for the CA computer. When you restore Certificate Services configuration, the disk volumes must implement the same drive letters. Disk volumes can be different sizes or implement different RAID levels, but the drive letters and locations must remain the same for theCA database, CA logs, CA configuration folder (if implemented), and operating system.
A copy of the CAPolicy.inf file deployed in the %windir% of the CA computer. The CAPolicy.inf must be in place when renewing the CA’s certificate.
Disaster Recovery Procedures:
There are two methods to backup and restore the Certification Authority. The methods are:
1- System State Backup
2- Certutil command line in combination of registry export
Update: It just came to my attention that System State Backup in Windows 2008 and 2008 R2 will not backup the private key of the CA. The private key will be stored in hidden folder structure "%systemdrive\ProgramData\Microsoft\Crypto\Keys" which will be linked and accessible via "%systemdrive%\users\all users\microsoft\crypto\keys". %systemdrive%\ProgramData\Microsoft\Crypto\Keys" is not included in System State backup as it's not in system writers metadata and so will be empty when doing a System State restore.
If you prefer to have System State Backup, then you should consider applying the following hotfix https://support.microsoft.com/kb/2603469 on your CAs running Windows Server 2008 or 2008 R2 to backup the Private Key.
Advantages and Disadvantages of Each Procedure:
Each method has advantages and disadvantages. The main advantage of System State backup is simplicity, where the administrator has to join an identical piece of hardware to the domain where the CA existed and restore System State Backup. The main disadvantage of System State is dependence on identical hardware
The Certutil command in combination with the registry export allows the administrator to restore the Certification Authority to any server – hardware agnostic. The main disadvantage of the Certutil command is the amount of steps required to perform the restore.
The document hereafter will only discuss the backup and restore methods using the Certutil command. In addition the document assumes Web Enrollment Pages are installed at the Certification Authority.
Back Up the Certification Authority:
Any proper backup of a CA should include the Certificate Security Protocol, Templates published at the CA, Private Key, Certificate Database and logs in addition to the configuration of the CA stored in HKLM\System\CurrentControlSet\Services\Certsvc\Configuration. The script below combines all of these steps
1- Log on as user who has CA administrator rights.
2- Create a folder under %Homedrive% called Backup.
3- Create a new text document under C:\scripts
4- Paste the following text:
Echo Backup Certification Authority, Certificates, Templates and CSP
c:
cd \scripts
Echo Y| del C:\backup\database
rd C:\backup\database
Echo Y| del c:\backup
Echo Backing up the Certification Authority and Certificates
certutil -backup -p Password c:\backup
Echo Backing up the registry keys
reg export HKLM\System\CurrentControlSet\Services\CertSvc\Configuration c:\backup\regkey.reg
Certutil –getreg CA\CSP > C:\Backup\CSP.txt
Echo Documenting all certificate templates published at the CA
Certutil –catemplates > C:\Backup\CATemplates.txt
*Note* You need to enter a valid Password in the script where noted. The immediate line following the Certification Authority backup will back up the registry
5- Save the file as “BackupCertificates.cmd”
6- Schedule a task to run every day using an administrative account.
7- Schedule your regular backup software job to backup the System State and the C:\Backup folder every day or copy the folder to a safe location.
Steps to Restore the Certification Authority:
Restoring the CA will require using the backup files taken from the Certification Authority, in addition to rebuilding a new server. The steps required are:
1- Extending the life of the CRL file
2- Decommission the Old Certification Authority
3- Install Active Directory Certificate Services (ADCS) at the new server
4- Restore the Certification Authority Configuration
5- Restore the Database and Templates to the Certification Authority
Extending the Life of the CRL file:
This step is necessary to ensure clients’ revocation files are processed in a timely manner,
1- Log on to a any machine in your domain as an administrator
2- Restore the CA's Private key to the machine
3- Obtain a CRL or certificate issued by the CA being tested. The CRL or certificate must correspond to the CA key and certificate being tested where you are restoring multiple keys.
4- Extend the life of the CRL by running Certutil –sign <CRLFileName.crl> ++dd, and when prompted , select the CA certificate (imported in the previous procedure) as the signing certificate.
Example:
Certutil -sign Contoso-Issuing-CA.crl ++03
5- Publish the CRL file to all distribution points as follows:
a. Copy the CRL file to the http distribution points
b. Log on to any machine in the domain as an enterprise admin and run the Certutil –f –dspublish <CRLFileName.crl>
You must now clean the keys from the test system. To clean the keys from the system
1. Log on as a member of local Administrators and delete the user profile of the administrator account using Advanced Properties in My Computer.
2. Delete the administrator account.
3. Securely erase unallocated areas of the disk to permanently remove traces of the keys by running the following command.
Cipher /W:%AllUsersProfile%
Note: Specifying %allusersprofile% as the path ensures that the cipher.exe command operates on the drive holding the user profiles. It clears the whole drive, not just the indicated path, hence making the machine unusable.
Decommission the Old Certification Authority:
This procedure is explained in details in a support article. Refer to https://support.microsoft.com/kb/889250 for the steps required to decommissions the old Certification Authority
Install Active Directory Certificate Services at the New Server:
The new server must have the same computer name as the old server. Furthermore, it should have the same Operating System of the failed server
1- Partition the server with the same volume names
2- Copy or restore the files from the Backup folder. You should have the database, PKCS12 (*.P12) file, the registry, CATemplates.txt, and CSP.txt to the new server.
3- In Server Manager, click Add Roles.
4- In the Select Role Services window, select Certification Authority and Certification Authority Web Enrollment – if installed previously , and then click Next
5- In the Specify CA Type dialog box, click the appropriate CA type based on the failed server CA type.
6- Click Use custom settings to generate the key pair and CA certificate, and then click Next.
7- In the Set Up Private Key windows, select Use existing private key and then select the option select a certificate and use its associated private key. Click Next
8- In the Select Existing Certificate, choose the Import option and browse to the PKCS12 file in the backup folder, type the password you used during the backup, and click OK, then click Install
9- Follow the setup screens until the CA is restored
Restore the Certification Authority Configuration:
1- Stop the Certificate Services service.
2- Locate the registry file that you restored, and then double-click it to import the registry settings. If the path that is shown in the registry export from the old CA differs from the new path, you must adjust your registry export accordingly. By default, the new path is C:\Windows in Windows Server 2003.
Restore the Database and Templates to the Certification Authority:
Use the Certification Authority snap-in to restore the CA database. To do this, follow these steps:
1- In the Certification Authority snap-in, right-click the CA name, click All Tasks, and then click Restore CA. The Certification Authority Restore Wizard starts.
2- Click Next
3- Click Certificate database and certificate database log.
4- Type the backup folder location, and then click Next.
5- Verify the backup settings. The Issued Log and Pending Requests settings should be displayed.
6- Click Finish, and then click Yes to restart Certificate Services when the CA database is restored.
7- In the Certification Authority snap-in, manually add or remove certificate templates based on the templates published at the CA using the CAtemplates.txt file
Comments
Anonymous
January 01, 2003
Yes, you can use the same script to backup a 2003 CAAnonymous
January 01, 2003
Yes! This was reported and verified by our engineersAnonymous
January 01, 2003
Ordeith, the script was updated. Thanks, AmerAnonymous
January 01, 2003
Thanks for blogging about the PKI, its really helpfull. one of the thing which i was trying to clarrify and not yet succeeded is when a CA (Issuing CA) is down how long can i survive. what i heard from folks around is that the certs will be working fine till the CRL expires. but will it work till the Certificate expire even if the CRL expired?Anonymous
January 01, 2003
Greg, You can archive the Private Key of the CA using Certutil -backupkey option and then use System State for your regular backup. One thing to keep in mind here, System State is hardware dependent, so you need to make sure you are restoring the CA on similar hardwareAnonymous
January 01, 2003
Hi Jake,
You don't have to restore the templates from backup nor do you have to recreate them. They are stored in AD. You just need to enable the certificate template for issuance by right clicking on certificate template, then clicking on new template to issue and finally selecting the certificate template.Anonymous
January 01, 2003
This depends on your distribution points, the certificate type used (if it checks for revocation) and the CRL period. For example, if your distribution point is LDAP, your certificates can access LDAP and already have a revocation time of 2 days left, then your issuing CA can be down for 2 days, until the client pulls downs a new CRL from the distribution point. Likewise for the HTTP distribution point. The most important thing to check is the Next Update in the CRL file itself because the client caches the CRL until the next update, and of course making sure the CRL file is already published at your distribution points for any client to pull it down. As for expired certificates, they will not be used, so it doesn’t matter if the CRL was expired or not.Anonymous
January 01, 2003
Hi AmerK, after we signing CRL by using following command, it won't change CRL Number, does it made any significance ? certutil -sign <existing CRL file name> <re-signed CRL file name>Anonymous
January 01, 2003
Hi Thanks for the post. Shall I use the same script to backup my Windows 2003 CA Regards LMSAnonymous
January 01, 2003
Does the CA have backups? If that is the case then you can restore it. You can always clean the CA records from Active Directory using support.microsoft.com/.../889250Anonymous
January 01, 2003
Hi Auro_vg There is no significance for the CRL number.Anonymous
January 01, 2003
Hi Auro, You need to modify the registry key if you restored the CA to a different computer name, and re-acl the CA's objects in Active Directory to make sure the CA can publish its CRL files. Step 12 in the article you referenced mentions you need to edit the registry keys. It is much simpler to restore the system to the same computer name.Anonymous
January 01, 2003
Hello !
Sorry for my bad english (i am french)
A question on publish crl in AD ...
I publish the crl of an offline ca root with : certutil -dspublish -f mycrlfile.crl srvcaroot (where srvcaroot is my netbios name of my server where is my caroot)
All is fine, and i saw the publication in the node sevices in the mmc "sites and services" under the cdp container (i have well a srvcaroot container created and mt crl in)
The probleme is that when i do a Gpupdate on a computer in the domain and see in the mmc certificate if the crl is coming down in the entreprise carevocation list folder nothing comes ... the revocation folder is not created at all
If i install ... manualy the revocation list on the computer of the domain ... the folder appears and the crl is in it
What is the problem ?
Help pleaseAnonymous
January 01, 2003
Hello AmerK, Can you please verify the following two points ? "The new server must have the same computer name as the old server. Furthermore, it should have the same Operating System of the failed server" But, technet.microsoft.com/.../cc755153(v=ws.10).aspx did not saying anything about this.
- As far my testing in lab am able to switch platforms and OS versions.
- I have changed target server name and restore PKI pvt. Key and DB - this gives me an option to fall back to original server by modifying dNSHostName attribute. what is your view ? Thanks Auro
Anonymous
January 01, 2003
Amerk,
One more scenario - in restoration/backup.
I have an two tier PKI - for Issuing CA we took a backup (from 2008R2) and restored into a new 2012 server with changed Server name.
published CRLs in old and new location everything is working as expected. all of a sudden we have observed OCSP servers are not able to pull OCSP certs. they are keep on looking for OLD Server name. and failing with event ID 34 (stating old Issuing CA Servername is not reachable RPC) and 23.
changed Registry entry @ HKLMSystemCCSServicesOcspSvcResponderReponderNameCAConfig and it is working back again.
JagadeeshAnonymous
January 01, 2003
This blog is up to date and applies to Windows Server 2008 and 2008 R2.Anonymous
June 22, 2010
One thing I am trying to get clarification on is what to do if the old CA is unavailable, like with a hardware failure. How does one "decommission" the old CA from AD if you can't boot the server?Anonymous
July 16, 2010
I have a question regarding the script, the row: Echo Y| del c:backup that is put after the creation of the CSP.txt and CATemplates.txt removes these two files, is there a point to create them in the first case then? I am probably lacking some deeper knowledge of CA:s but I would appreciate a clarification. Thank you ! Best regards /OrdeithAnonymous
July 26, 2010
Just to be crystal clear. By saying that the Windows Server 2008 and 2008R2 system state backup does not backup the private keys of the CA, You are saying that Brian Komars Book "Windows Server 2008 PKI and Certificate Security" is wrong on this? (Page 310)Anonymous
August 11, 2010
Per TechieGirl, we just ran into a CA crash where we found out we did not have the entire CA backed up. So we had to "decommission' the CA, but there are a number of blog entries on this (see support.microsoft.com/.../555151 for some documentation). One key point is removing the CA entries from the ActiveDirectory containers. Keep in mind that, once you do that, AD will automaticaly remove the CA root cert from every client computer. The net effect is that any older certificates you had (and want to replace with the new CA) are immediately invalidated. This means no more email signing, no certificate-based VPN connections (if you use that instead of PSK), etc. Be sure that's what you want and that (if possible) you've setup the new CA and updated all client computers (and email security) before removing the old CA entries from AD sites and services. If you are not using automatic CRL, you can continue using the old certficates from the crashed CA until they expire as long as the crashed CA remains in AD sites and services. Just be aware and choose wisely...Anonymous
December 21, 2010
certcollection.org has an exact copy of this blog post here: certcollection.org/.../53883-disaster-recovery-procedures-for-active-directory-certificate-services-adcsAnonymous
September 14, 2011
Hello, are there any windows updates that the private keys will store in future (2008 R) in the systemstate backup or is this block up to date what the backup of private keys belongs? i have done the points, i wonder that the Database Directory has 17 MB, after a new run 18 MB. every backup i made the database directory grow about 1 MB. Why? Regards EaxAnonymous
November 30, 2011
Can't I just archive the private key from my Enterprise CA and use System State to back up the CA going forward? I'd imagine that, after a restore of system-state (after a failure), I'd then have to import the private key but that shouldn't be a problem, right? Thanks for a great guide!Anonymous
November 30, 2011
Amerk Thanks for the quick response. That's the beauty of virtualization! Thanks much! Regards, GregAnonymous
May 16, 2012
Hi Amerk, Is the backup script worked for "Standalone root CA"? I try to perform "Certutil –catemplates > C:BackupCATemplates.txt". However, it returned the error. Regards, TerryAnonymous
March 24, 2014
Certificate services was inadvertently uninstalled this morning. We promptly restored the system state from the previous night backup. After a reboot the certificate services showed back up and all the certificates showed as well.
If I click on certificate templates in the CA snapin I get an error and no templates are showing.
I checked ADSI and the server is showing in all relevant PKI areas accept the templates area. I'm assuming that this is the issue
I'm am a novice on certificates. As of right now we have had no issues but I am concerned that this will change in the next 24-48 hours and I'd like to cut it off at the pass.
After restoring the system state without reinstalling certificate services what concerns should I be aware of? I did backup the CA as soon as it came back up after the restore of the system state.
Do I still need to reinstall certificate services and restore from the CA backup?
Can I manually just create the object in ADSI?
Please help
Thanks
JakeAnonymous
March 24, 2014
The comment has been removedAnonymous
October 11, 2014
I am working to setup a colo site. my question is - Do i need a cert authority server at my DR site ?Anonymous
September 15, 2015
Hey Everyone, welcome to my blog and my very first blog post :)
Public Key Infrastructure is myAnonymous
June 28, 2016
Hi AllI have a question about the contents of the catemplates.txt file.I have CA Administrator rights however when i run the script (or just a certutil -catemplates) my file contains just 13 lines. it's basically a list of cert templates configured as follows:"templateshortname:template description -- Auto-Enroll: Access is denied."Is it solely run to create a list, or should it contain the configs to set up the certs? I ask because i do maintain a custom template but i went through what i could read of all the files and couldnt' see where the configurations for the templates was being stored.- Anonymous
February 23, 2017
Correct, it is only to get a list of the certs, not provide the settings inside them, etc. The template themselves are backed with Active Directory.
- Anonymous
Anonymous
March 28, 2017
Hi Amerk, Sorry its a little different question from topic . We are migrating from 2003 R2 x64 to 2008 Sp2 x64. Can we do it ? (Both machines are existing Domain Controllers) I read in a MS article that 2008 should be R2.2nd question: As per my understanding I need to maintain same CA name but host name can differ. where will I see an option to chose CA name on 2008 DC. Please advise. Any help will be appreciatedI found this simple article : https://blogs.technet.microsoft.com/canitpro/2014/11/11/step-by-step-migrating-the-active-directory-certificate-service-from-windows-server-2003-to-2012-r2/#comment-193195Anonymous
May 01, 2018
I have a 2008 root ca that died with no backups in place before my time and am going through the process of implementing a side by side for the sake of the issued certs. I want to remove the old templates and export them to the new root CA, then remove the templates from the old so it no longer issues new certificates. Will these instructions do this or will I need some other tool or script to make this work? Or can I recreate the root without a backup by exporting the root cert from the subordinate?- Anonymous
May 04, 2018
No, you cannot obtain root private key from the subordinate. CA templates are stored in the configuration partition of AD, so you can just remove from the old root and add to the new root. However, it sounds like you are utilizing an Enterprise Root (online root) and is not recommended. See the following article for deploying a 2-tier PKI with an offline root (standalone):https://social.technet.microsoft.com/wiki/contents/articles/15037.ad-cs-step-by-step-guide-two-tier-pki-hierarchy-deployment.aspx- Anonymous
May 07, 2018
Chris the old CA died and I am attempting to recreate the PKI infrastructure and everything I have read says to export the old templates and import them. There is however no explanation on how to go about this process since I do not have a backup copy of the old root CA I have no way of restoring that specifically. So if I am reading your explanation correctly I don't have to export and import the templates because they are stored in AD?
- Anonymous
- Anonymous