Exchange 2010 SP1: StartDagServerMaintenance.ps1 fails when a server contains databases with a single copy.
In Exchange 2010 Service Pack 1 we introduced some new DAG management scripts. These scripts can be found in the Exchange Server installation directory \ scripts. (This is usually c:\Program Files\Microsoft\Exchange Server\v14\scripts).
One of the scripts introduced is the StartDagServerMaintenance.ps1 script. More information on this script can be found at:
https://technet.microsoft.com/en-us/library/ff625233.aspx
https://technet.microsoft.com/en-us/library/dd298065.aspx
When administrators utilize this script the following actions are being taken:
1) All database copies are moved to another server in the DAG based on the selection of the next best copy.
2) If the cluster core resources are owned on the node the resources are arbitrated to a different DAG member (thereby moving the Primary Active Manager functionality to another node).
3) The DatabaseCopyAutoActivationPolicy property of the mailbox server is set to a value of BLOCKED thereby preventing the DAG member from receiving or activating database copies.
4) The individual database copies hosted on the DAG member are activation suspended.
5) The node is paused within the cluster service preventing the cluster core resources from arbitrating to the node (and thereby preventing the node from becoming the Primary Active Manager).
When utilizing a DAG it is not necessary to replicate all databases that exist on DAG members. It is not uncommon to have standalone databases (databases that are on a DAG member but not replicated to another member) present on a member where the StartDagServerMaintenance.ps1 script will be utilized. Unfortunately when utilizing the script in its current form in this configuration the script fails to complete its tasks and cannot completely put the node into maintenance mode. (Only databases are successfully moved off the member).
The administrator may note the following when executing the script on a member that contains a single database copy:
[PS] C:\Program Files\Microsoft\Exchange Server\V14\Scripts>.\StartDagServerMaintenance.ps1 -serverName DAG-1
The following objects are hosted by 'DAG-1', before attempting to move them off: `n(Primary Active Manager=DAG-1) (Mailbox='Discovery Search Mailbox', Reason='Mailbox is hosted on 'DAG-1-DB0', which is not a replicated database. ) (Mailbox='Journal Internal', Reason='Mailbox is hosted on 'DAG-1-DB0', which is not a replicated database. ) (Mailbox='MicrosoftExchange Approval Assistant', Reason='Arbitration Mailbox is hosted on 'DAG-1-DB0', which is not a replicated database.) (Database='DAG-DB0', Reason='Copy is active'))
Write-Error : The following objects are still hosted by 'DAG-1', even after attempting to move them off: `n(Mailbox='Discovery Search Mailbox', Reason='Mailbox is hosted on 'DAG-1-DB0', which is not a replicated database. ) (Mailbox='Journal Internal', Reason='Mailbox is hosted on 'DAG-1-DB0', which is not a replicated database. ) (Mailbox='Microsoft Exchange Approval Assistant', Reason='Arbitration Mailbox is hosted on 'DAG-1-DB0', which is not a replicated database. ))
At C:\Program Files\Microsoft\Exchange Server\V14\Scripts\StartDagServerMaintenance.ps1:216 char:16
+ write-error <<<< ($StartDagServerMaintenance_LocalizedStrings.res_0014 -f ( PrintCriticalMailboxResourcesOutput($criticalMailboxResources)),$shortServerName) -erroraction:stop
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Microsoft.PowerShell.Commands.WriteErrorCommand
If an administrator encounters this condition the following process can be utilized to place the DAG member into maintenance mode. (In our example server DAG-1 in the DAG named “DAG” [pretty creative eh?] is the server we will be placing in maintenance mode)
1) Execute a get-mailboxdatabasecopystatus * and verify that at least one other non-lagged copy of each replicated database is healthy.
[PS] C:\>Get-MailboxDatabaseCopyStatus *
Name Status CopyQueue ReplayQueue LastInspectedLogTime ContentIndex
Length Length State
---- ------ --------- ----------- -------------------- ------------
DAG-1-DB0\DAG-1 Mounted 0 0 Healthy
DAG-DB0\DAG-1 Mounted 0 0 Healthy
DAG-DB1\DAG-1 Healthy 0 0 7/13/2011 8:22:55 AM Healthy
DAG-2-DB0\DAG-2 Mounted 0 0 Healthy
DAG-DB1\DAG-2 Mounted 0 0 Healthy
DAG-DB0\DAG-2 Healthy 0 4 7/13/2011 8:48:34 AM Healthy
DAG-DB0\DAG-3 Healthy 0 147 7/13/2011 8:48:34 AM Healthy
DAG-DB1\DAG-3 Healthy 0 140 7/13/2011 8:22:55 AM Healthy
DAG-DB0\DAG-4 Healthy 0 409 7/13/2011 8:48:34 AM Healthy
DAG-DB1\DAG-4 Healthy 0 307 7/13/2011 8:22:55 AM Healthy
MBX-1-DB0\MBX-1 Mounted 0 0 Healthy
MBX-1-RDB\MBX-1 Mounted 0 0 Healthy
2) Execute a move of all active database copies off the server. This can be done with the command move-activemailboxdatabase –server <MaintenanceServer> (Note: No target server is specified which means the next best copy will be automatically selected for activation)
[PS] C:\>Move-ActiveMailboxDatabase -Server DAG-1 -Confirm:$FALSE
Identity ActiveServerAtS ActiveServerAtE Status NumberOfLogsLost RecoveryPoint MountStatus MountStatus
tart nd Objective AtMoveStart AtMoveEnd
-------- --------------- --------------- ------ ---------------- ------------- ----------- -----------
DAG-1-DB1 dag-1 dag-1 Warning Mounted Mounted
DAG-1-DB0 dag-1 dag-1 Warning Mounted Mounted
DAG-DB0 dag-1 dag-2 Succeeded 0 7/13/2011 8:5 Mounted Mounted
3:24 AM
WARNING: An Active Manager operation failed. Error: The database action failed. Error: You cannot perform a switchover
operation on database 'DAG-1-DB1' because the database is not configured for replication.. [Database: DAG-1-DB1,
Server: DAG-1.domain.com]
WARNING: An Active Manager operation failed. Error: The database action failed. Error: You cannot perform a switchover
operation on database 'DAG-1-DB0' because the database is not configured for replication.. [Database: DAG-1-DB0,
Server: DAG-1.domain.com]
3) Move the cluster core resources to another node within the DAG. This can be accomplished using the command cluster.exe <DAGFQDN> group “Cluster Group” /moveto:<NODE>
[PS] C:\>cluster DAG.domain.com group "Cluster Group" /moveto:DAG-2
Moving resource group 'Cluster Group'...
Group Node Status
-------------------- --------------- ------
Cluster Group DAG-2 Online
4) Pause the node within the cluster. This can be done utilizing the command cluster.exe <DAGFQDN> node <NODENAME> /pause
[PS] C:\>cluster DAG.domain.com node DAG-1 /pause
Pausing node 'DAG-1'...
Node Node ID Status
-------------- ------- ---------------------
DAG-1 1 Paused
5) Set the DatabaseCopyAutoActivationPolicy of the server to BLOCKED. This can be done using the command set-mailboxserver –identity <DAGMember> –databasecopyautoactivationpolicy:BLOCKED
[PS] C:\>Set-MailboxServer -Identity DAG-1 -DatabaseCopyAutoActivationPolicy:BLOCKED
6) Suspend all individual copies for activation. This can be done using the command get-mailboxdatabasecopystatus *\<DAGMember> | suspend-mailboxdatabasecopy –activationOnly:$TRUE
[PS] C:\>Get-MailboxDatabaseCopyStatus *\DAG-1 | Suspend-MailboxDatabaseCopy -ActivationOnly:$TRUE
Database "DAG-1-DB0\DAG-1" has only one copy. This task is supported only for databases that have more than one copy.
+ CategoryInfo : InvalidOperation: (DAG-1-DB0:ADObjectId) [Suspend-MailboxDatabaseCopy], InvalidOperation
Exception
+ FullyQualifiedErrorId : 7325D1AB,Microsoft.Exchange.Management.SystemConfigurationTasks.SuspendDatabaseCopy
Confirm
Are you sure you want to perform this action?
Suspending activation of mailbox database copy "DAG-DB0" on server "DAG-1".
[Y] Yes [A] Yes to All [N] No [L] No to All [?] Help (default is "Y"): a
At this time it should be safe for the administrator to perform DAG server maintenance. When the maintenance is complete the script StopDagServerMaintenance.ps1 can be utilized to take the DAG member out of maintenance mode.
Comments
Anonymous
January 01, 2003
And this will be the result if you have e.g. Public Folder database in DAG member?Anonymous
January 01, 2003
@Makol...
We're you attempting to put multiple servers into maintenance mode at the same time. If so - you should probably only be doing one server at a time.
Also - do you have databases that only contain two copies?
TIMMCMICAnonymous
January 01, 2003
@DougZuck Great question - here's my take on it. Juts because a server is in maintenance mode does not mean you do not want it shipping and replaying logs. This allows the server to continue to replay logs and be up to date so that when you come out of maintenance mode it is ready to go. If the patch happens to be exchange, replay and shipping is inherently stopped when the replication service and information store service stop. Suspending replication would increase the maintenance time overall if a lot of logs were generated during the maintenance period. Can you suspend log shipping and replay - absolutely. It's just not something that's done as a normal course when putting a node in mainteance mode. TIMMCMICAnonymous
January 01, 2003
Great article. However, one thing that I'm not clear on is why is it sufficient to simply suspend database copies from activation? Wouldn't we want to suspend replication altogether to the copies on the server in question? If we're going to be performing software and/or hardware maintenance on the server with multiple reboots and service starts and stops, wouldn't it be safer to suspend replication rather than just block/suspend activation?Anonymous
January 01, 2003
@Woter... Thanks TIMMCMICAnonymous
January 01, 2003
Tim - thanks for confirming. This makes complete sense. -DougAnonymous
January 01, 2003
@Petri X In my testing I did not see where this issue is exposed by a public folder store. TIMMCMICAnonymous
January 25, 2012
Nice article !! Thanks for Sharing Tim :)Anonymous
September 27, 2012
I understand the inibillity for the script to work on a two node DAG was fixed in SP2. It appears this is not the case. We are SP2 RU2 and the problems described are apparent. ThanksAnonymous
September 27, 2012
Update: Actually, it does work in SP2. Very well, however it does not suspend the copy as I was lead to believe. TIMMCMIC gives a good reason why above.Anonymous
March 14, 2014
Hi all
Thank you for this detailed description.
Our DAG containing 3 nodes I faced exactly the same issue when I tryed to bring the second server into maintenance mode.
So I processes all the steps as described here. They run through without an error, but I was still not possible to bring the second standby server into maintenace mode. I still got the same errors.
Tanks.Anonymous
March 15, 2014
Hi Timmcmic
Ok let me explain the situation.
We have a 3 node DAG. In addition we do have 2 servers which run the CAS and Hubtransport role (CAS array).
All Exchange servers are virtual (ESX). I know that it is not suggested to run the MBX servers as virtual servers, but newer the less. When I joined the company it was a fact. And it is still one.
I am aware that it is suggested to bring only one server in maintenance mode at a time.
But due to a network maintenace this weekend I had to shut down all Exchange servers. That is why I had to shut down the whole DAG. So what would be your suggestion to shut down the whole DAG without bring two of the servers in maintenance mode first?
Your answer is verry appreciated.
Thanks.