Freigeben über

Part 5: Datacenter Activation Coordination: How do I force automount consensus?

  • Artikel

In order for any database availability group (DAG) that has Datacenter Activation Coordination (DAC) mode to be able to mount any databases, the DAG itself much reach automount consensus. In the following blog posts, I discuss and outline situations where automount consensus may not be achieved, resulting in dismounted databases.  A DAG in DAC mode which cannot achieve automount consensus will not automatically mount databases, and administrators are blocked from manually mounting them, as well.

 

Let’s look at an example to illustrate this scenario.  In this example, I have a 4-member DAG comprised of servers DAG-1, DAG-2, DAG-3, and DAG-4

 

image

 

DAG-4 experiences a hardware failure, goes down, and is scheduled for replacement.  Although the server is offline, it is still a member of the DAG and it’s underlying cluster.

 

image

 

Next, a power failure occurs resulting in DAG-1, DAG-2, and DAG-3 going offline.

 

image

 

After power is restored, DAG-1, DAG-2, and DAG-3 are booted up.

 

image

 

The cluster service is successfully initialized and a cluster is formed.  You can verify the nodes are operational by running

 

Windows 2008:

PS C:\> cluster node
Listing status for all available nodes:

Node Node ID Status
-------------- ------- ---------------------
DAG-4 1 Down
DAG-3 2 Up
DAG-2 3 Up
DAG-1 4 Up

 

 

Windows 2008 R2:

PS C:\> Import-Module FailoverClusters

PS C:\> Get-ClusterNode | fl name,state

Name : DAG-1
State : Up

Name : DAG-2
State : Up

Name : DAG-3
State : Up

Name : DAG-4
State : Down

 

Using the Get-MailboxDatabase cmdlet, we can see that, even though the servers are up and the cluster is running, no databases are mounted:

 

[PS] C:\>Get-MailboxDatabase -Status | fl name,mounted

Name : Mailbox Database 1252068500
Mounted : False

Name : Mailbox Database 1757981393
Mounted : False

Name : Mailbox Database 1370762657
Mounted : False

Name : Mailbox Database 1511135053
Mounted : False

 

Attempting to mount a database (with or without the –Force parameter) results in the following error:

 

[PS] C:\>Mount-Database "Mailbox Database 1252068500"
Couldn't mount the database that you specified. Specified database: Mailbox Database 1252068500; Error code: An Active Manager operation failed. Error An Active Manager operation encountered an error. To perform this operation, the server must be a member of a database availability group, and the database availability group must have quorum. Error: Automount consensus not reached.. [Server: DAG-1.exchange.msft].
+ CategoryInfo : InvalidOperation: (Mailbox Database 1252068500:ADObjectId) [Mount-Database], InvalidOperationException
+ FullyQualifiedErrorId : FD5FDFE8,Microsoft.Exchange.Management.SystemConfigurationTasks.MountDatabase

So what needs to be done to achieve automount consensus and mount the databases?  It’s simple: the administrator can run Start-DatabaseAvailabilityGroup.  The cmdlet can be run using the –MailboxServer parameter to specify one of the mailbox servers that is functional.  After the cmdlet is run, the specified server will have its DACP bit set to 1.  The other servers in the DAG will see that the DACP bit on the specified server is set to 1 and this will cause them to set their DACP bits to 1.  And once all of the DAG members have DACP set to 1, automount consensus is reached, and the databases can be mounted. Here is an example:

 

[PS] C:\>Start-DatabaseAvailabilityGroup -Identity DAG -MailboxServer MBX-1

[PS] C:\>Get-MailboxDatabase -Status | fl name,mounted

Name : Mailbox Database 1252068500
Mounted : True

Name : Mailbox Database 1757981393
Mounted : True

Name : Mailbox Database 1370762657
Mounted : True

Name : Mailbox Database 1511135053
Mounted : True

 

Start-DatabaseAvailabilityGroup can also be used in scenarios where the remaining survivor of a DAG failure is a single member that is using the witness server boot time to achieve automount consensus.  In this case the cmdlet forces the member’s DACP bit to 1 and updates the boot times in the registry for future restarts.

 

As a side note of caution, administrators should be careful when using Start-DatabaseAvailabilityGroup.  It is possible that a split brain condition could be created by using this cmdlet to start the wrong DAG members, or it by using it at the wrong stage of a datacenter switchover. To ensure your datacenter switchovers go smoothly, I recommend using the recently released Datacenter Switchover Tool.

 

========================================================

Datacenter Activation Coordination Series:

 

Part 1:  My databases do not mount automatically after I enabled Datacenter Activation Coordination (https://aka.ms/F6k65e)
Part 2:  Datacenter Activation Coordination and the File Share Witness (https://aka.ms/Wsesft)
Part 3:  Datacenter Activation Coordination and the Single Node Cluster (https://aka.ms/N3ktdy)
Part 4:  Datacenter Activation Coordination and the Prevention of Split Brain (https://aka.ms/C13ptq)
Part 5:  Datacenter Activation Coordination:  How do I Force Automount Concensus? (https://aka.ms/T5sgqa)
Part 6:  Datacenter Activation Coordination:  Who has a say?  (https://aka.ms/W51h6n)
Part 7:  Datacenter Activation Coordination:  When to run start-databaseavailabilitygroup to bring members back into the DAG after a datacenter switchover.  (https://aka.ms/Oieqqp)
Part 8:  Datacenter Activation Coordination:  Stop!  In the Name of DAG... (https://aka.ms/Uzogbq)
Part 9:  Datacenter Activation Coordination:  An error cause a change in the current set of domain controllers (https://aka.ms/Qlt035)

========================================================

Comments

  • Anonymous
    January 01, 2003
    thanks!

  • Anonymous
    January 01, 2003
    Thanks!

  • Anonymous
    January 01, 2003
    @arvis

    I'm glad you enjoyed it.

    TIMMCMIC

  • Anonymous
    January 23, 2015
    The comment has been removed

  • Anonymous
    April 27, 2015
    Thanks & great article

  • Anonymous
    September 11, 2015
    Thanks Buddy. That was awesome.

  • Anonymous
    October 18, 2015
    @Simon...

    Glad it was helpful.

    TIMMCMIC

  • Anonymous
    February 15, 2016
    Thanks a lot. Really helpful post.

    I have one DAG spread across two branches. In each branch there is a single DAG member, and one file server witness in one of the branches. Due to power constraints we shut down the servers in each location daily, then bring them back on the next day, except during weekends when only one of the offices would be open.
    I would expect the DAG member in this branch to have quorum since it is in the same location as the file server witness, but this is not so. The Database always starts up Dismounted, making me issue the Start-DatabaseAvailabilityGroup command every weekend. Please is this how it is supposed to be? Or am I missing something?