Verifying the file share witness server / directory in use for Exchange 2010
If you’ve read my blog post on file share witness oddities you might be asking yourself “how can I actually verify what file share witness is in use in my environment?”
There are three different ways to verify the witness in use for Exchange 2010.
1) Exchange Management Shell
The Get-databaseavailabilitygroup –status command returns all the settings for the DAG from Active Directory. By adding the –status switch we also query additional values from the Cluster service and Replication service. These are not queried by default as they can delay the command from returning normal configuration values. Here is an example of the output:
[PS] C:\>Get-DatabaseAvailabilityGroup -Identity DAG -Status | fl
RunspaceId : 717eb01d-a17b-4e8e-b018-acf00a0d748d
Name : DAG Servers : {DAG-4, DAG-3, DAG-2, DAG-1}
WitnessServer : mbx-1.domain.com
WitnessDirectory : c:\DAG-FSW
AlternateWitnessServer : mbx-2.domain.com
AlternateWitnessDirectory : c:\DAG-FSW
NetworkCompression : Enabled
NetworkEncryption : Enabled
DatacenterActivationMode : DagOnly
StoppedMailboxServers : {}
StartedMailboxServers : {DAG-3.domain.com, DAG-4.domain.com, DAG-2.domain.com, DAG-1.domain.com}
DatabaseAvailabilityGroupIpv4Addresses : {10.0.0.24}
DatabaseAvailabilityGroupIpAddresses : {10.0.0.24}
AllowCrossSiteRpcClientAccess : False
OperationalServers : {DAG-1, DAG-2, DAG-4, DAG-3}
PrimaryActiveManager : DAG-1
ServersInMaintenance : {}
ThirdPartyReplication : Disabled
ReplicationPort : 64327
NetworkNames : {DAG-4-iSCSI, DAG-MAPI, DAG-REPL-A, DAG-REPL-B}
WitnessShareInUse : Primary
AdminDisplayName :
ExchangeVersion : 0.10 (14.0.100.0)
DistinguishedName : CN=DAG,CN=Database Availability Groups,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=domain Home,CN=Microsoft Exchange`,CN=Services,CN=Configuration,DC=home,DC=domain,DC=com
Identity : DAG
Guid : 72c87136-6721-46e6-ac43-2ad5f6bd66d2
ObjectCategory : domain.com/Configuration/Schema/ms-Exch-MDB-Availability-Group
ObjectClass : {top, msExchMDBAvailabilityGroup}
WhenChanged : 1/29/2012 5:34:25 PM
WhenCreated : 9/19/2009 6:16:52 PM
WhenChangedUTC : 1/29/2012 10:34:25 PM
WhenCreatedUTC : 9/19/2009 10:16:52 PM
OrganizationId :
OriginatingServer : DC-5.domain.com
IsValid : True
In this example you can see the attribute WitnessShareInUse with a value of Primary. This lets the administrator know that the current witness configured for cluster use is the primary file share witness (in this case the witness server and witness directory).
[PS] C:\>Get-DatabaseAvailabilityGroup -Identity DAG -Status | fl name,servers,witnessserver,witnessdirectory,alternatew
itnessserver,alternatewitnessdirectory,operationalservers,primaryactivemanager,witnessshareinuse
Name : DAG
Servers : {DAG-4, DAG-3, DAG-2, DAG-1}
WitnessServer : mbx-1.domain.com
WitnessDirectory : c:\DAG-FSW
AlternateWitnessServer : mbx-2.domain.com
AlternateWitnessDirectory : c:\DAG-FSW
OperationalServers : {DAG-1, DAG-2, DAG-4, DAG-3}
PrimaryActiveManager : DAG-1
WitnessShareInUse : Alternate
In this example you can see the attribute WitnessShareInUse with a value of Alternate. This is an example of where the AlternateWitnessServer and AlternateWitnessDirectory are configured for cluster use.
[PS] C:\>Get-DatabaseAvailabilityGroup -Identity DAG -Status | fl name,servers,witnessserver,witnessdirectory,alternatew
itnessserver,alternatewitnessdirectory,operationalservers,primaryactivemanager,witnessshareinuse
WARNING: The witness server and directory currently in use by database availability group 'DAG' doesn't match the
configured primary or alternate witness server. This may be due to Active Directory replication latency. If this
condition persists, please use the Set-DatabaseAvailabilityGroup cmdlet to correct the configuration.
Name : DAG
Servers : {DAG-4, DAG-3, DAG-2, DAG-1}
WitnessServer : mbx-1.domain.com
WitnessDirectory : c:\DAG-FSW
AlternateWitnessServer : mbx-2.domain.com
AlternateWitnessDirectory : c:\DAG-FSW
OperationalServers : {DAG-1, DAG-2, DAG-4, DAG-3}
PrimaryActiveManager : DAG-1
WitnessShareInUse : InvalidConfiguration
In this example you can see the attribute WitnessShareInUse with a value of InvalidConfiguration. There is also a warning displayed indicating that the witness server in use does not match either the primary or alternate witness. This is an indication that the file share witness was modified outside of Exchange and the settings currently in use are not correct. Administrators can correct this by running the set-databaseavailabilitygroup command.
2) Utilize cluster commands
Windows 2008 / Windows 2008 R2
Using the command prompt execute the cluster <DAGNAME> res command. This will output all the resources within the cluster.
[PS] C:\>cluster dag.domain.com res
Listing status for all available resources:
Resource Group Node Status
-------------------- -------------------- --------------- ------
Cluster IP Address Cluster Group DAG-1 Online
Cluster Name Cluster Group DAG-1 Online
File Share Witness Cluster Group DAG-1 Online
Highlighted in red you can see the display name of the File Share Witness resource. With this information you can run the command cluster <DAGNAME> res “file share witness display name” /priv.
[PS] C:\>cluster dag.domain.com res "File Share Witness" /priv
Listing private properties for 'File Share Witness':
T Resource Name Value
-- -------------------- ------------------------------ -----------------------
S File Share Witness SharePath \\mbx-1.domain.com\DAG.domain.com
D File Share Witness ArbitrationDelay 6 (0x6)
This command lists the private properties within cluster associated with the file share witness resource. In our case we are interested in the SharePath. According to this output the current file share witness server is MBX-1.
Windows 2008 R2
Using powershell import the FailoverClusters modules.
[PS] C:\>Import-Module FailoverClusters
Issue the command Get-ClusterQuorum –cluster <DAGNAME> | fl
[PS] C:\>Get-ClusterQuorum -Cluster DAG.home.e-mcmichael.com | fl
Cluster : DAG
QuorumResource : File Share Witness
QuorumType : NodeAndFileShareMajority
Highlighted in red is the display name of the file share witness resource. Using the command Get-ClusterResource “Display Name” –cluster <DAGNAME> | Get-ClusterParameter
[PS] C:\>Get-ClusterResource "File Share Witness" -Cluster DAG.home.e-mcmichael.com | Get-ClusterParameter
Object Name Value Type
------ ---- ----- ----
File Share Witness SharePath \\mbx-1.domain.c... String
File Share Witness ArbitrationDelay 6 UInt32
This command lists the private properties within cluster associated with the file share witness resource. In our case we are interested in the SharePath. According to this output the current file share witness server is MBX-1.
3) Failover Cluster Manager
Using Failover Cluster Manager connect to the cluster service. You can connect to either a node or specify the DAG name as the connection point.
Click on the cluster name in the upper left hand corner of the utility.
In the center window information is displayed regarding the cluster configuration.
One piece of information is the “Quorum Configuration”. This will list the type of quorum in use and if a file share witness is configured the server and share name utilized as the witness.
In this example you can see that the cluster is configured for a quorum type of Node and File Share Majority with the file share witness server MBX-1.
Comments
Anonymous
January 01, 2003
@Erick:
I'm glad you found it helpful.
TIMMCMICAnonymous
January 01, 2003
@sophal...
File share witness settings are managed with set-databaseavailabilitygroup.
TIMMCMICAnonymous
January 01, 2003
@kid...
The file share witness is present on each DAG. It's there to support the dynamic removal and addition of nodes.
TIMMCMICAnonymous
January 01, 2003
@Red:
I do not have the commands handy - most customers I work with do not change it once referring to the correct method to validate witness in use.
The last time I corrected the display name we converted the cluster to node majority quorum type and then ran set-databaseavailabilitygroup which reset the witness settings.
TIMMCMICAnonymous
January 01, 2003
@Sunil...
First and foremost this customer needs to give up on this "requirement". Second - we need to understand how the cluster group plays into quorum. Quorum is always V/2+1. Regardless of where the cluster group is housed, there is only one site that will have quorum. The cluster group has no impact on that in this scenario. If the cluster group is housed in DR, and the WAN goes down, it will be arbitrated to the site with 3 votes and will stay online.
WE DO NOT support attempting to alter the arbitration of the cluster group preventing it from fully migrating to each node...so you will not see any recommendations of the like here.
Also - to your backup comment. The backup software should only ever be contacting the DAG name in order to determine node / database states. It should never be pulling data from that name. In theory if properly designed, the backup solution should be able to contact the name on either of the available IP addresses, and should move forward with individual node backups from that point forward. Again - no reason to be stressed about where the cluster core resource are.
TIMMCMICAnonymous
January 01, 2003
@Sunil:
The best summary that I've found is located here:
http://support.microsoft.com/kb/299631
I guess I don't necessarily spend much time looking at it - because it should not matter. IE - you should not be investing this much time into where the group goes - because it should not matter.
To your second point - maybe instead of worrying about where the group goes the customer should be focusing on the firewall issues which prevent the solution from effectively working the way it should.
:-)
TIMMCMICAnonymous
January 01, 2003
@Sunil...
The cluster service will pick an available node to arbitrate the cluster core resources to - and that process is not aware of location.
You really should not care that the cluster group is owned on a server in the DR site.
TIMMCMICAnonymous
November 01, 2012
What is the value of 'witnessinuse" when there are odd number of nodes in DAG/Cluster ?Anonymous
March 12, 2014
The comment has been removedAnonymous
June 07, 2014
@NathanScott -- i know you asked this along time ago but the answer is none. there will not be a witnessserverinuse when the nodes are odd. think about it, the vicepresident (witness) doesn't get to vote unless there is a tie in the senate (cluster). and yes, i did just call the senate a cluster!Anonymous
August 21, 2014
in this post has only show the status so how to configure alternate file share witness?Anonymous
September 18, 2014
Hi,
who can explain about witness server/ witness directory on Exchange server? When I am add New DAG it ask about that.Anonymous
December 18, 2014
Thank you for sharing... I did not know about it and had craze to know.. It is really useful...Anonymous
January 31, 2015
We have a 4 node DAG cluster running on Windows 2008 R2. Two nodes and file share witness in Production site and remaining two nodes in DR site. If we stop cluster service on any of the node in DC site. Cluster group gets failover to DR site. why is cluster group not going to second node in DC site, (as we have Node 2 and FSW in production site)? why is it going to DR (there also we have two votes?Anonymous
January 31, 2015
Thanks a lot for your quick response.
We have a 4 node Windows 2008R2 cluster running with Exchange DAG. Two production nodes are in DC site and other two are in DR site. We have a file share witness configured in DC site, quorum model is Node with file share majority. We have Node1, Node2 and FS three votes in DC and Node3, Node4 (two votes) in DR site. We have two exchange DB copies active (healthy) on two production nodes and DR copies are healthy but not active.
We have a requirement to have cluster group online always in DC nodes. But the problem here is, cluster group always gets failed over to DR site (Node1/2 to Node3/4). We don’t see any network issues between DR and DC sites. To isolate N/W issues, we have done below steps.
1. Stopped cluster service and disabled NICs on NODE1, Immediately cluster group got failed over to DR NODE4.
2. If we stop cluster service on DR NODE4, it’s back to DC Node2. (this is the standard behavior as we have 3 votes in DC and 1 Vote in DR (when we stopped cluster service on Node4).
3. When we stop cluster service on NODE 1 (DC), still cluster have two more VOTES (NODE 2 and File Share) in same production site). Why it it getting failover to DR site?
If both sites has equal amount of votes how cluster will get formed and who will own the cluster group?
Customer question is if we stop on Node1, it gets failover to DR (Node4) even though you have Node2 and FSW in DC. if you stop in DR, it will come to DC.
They want to understand how cluster group is getting moved (initially thought Node ID's). When we stop cluster service on node1 (DC), it went to node4 (DR) highest node ID 4. If we stop on Node 4 (DR) it should come to next height Node ID 3 (which is DR) not happening. Pls help us to understand this....
they want DAG (Cluster group) always in DC. (If there is no network issues/if you stop clussvc on any one node in DC)...Appreciate your quick help.Anonymous
January 31, 2015
Why they are worrying if cluster group is in DR site, is they take backup of DB's with DAG name, if Cluster group (DAG) is in DR they can't take backup...so they want always it to be on DC. They want to know the root cause how why cluster group is moving to DR..(2 nodes in DC, 2 in DR, 1 FSW in DC)..Anonymous
January 31, 2015
Thanks for your time and explanation Tim, really helps. Let me put my question and understanding in direct.
1. When I have Two Nodes (Node1, Node2) and FSW in Production site (3votes) and Two nodes (Node3, Node4) in DR site (2 votes). Possible owners of cluster group are (N1, N2, N3, N4). If I stop a cluster service / Disable a NIC on Node1 in Production Site, 4/2+1=3 Still my DAG can form quorum with remaining 3 nodes (N2, N3 and N4) no FSW needed here to form a quorum. I understand next node will be selected based on cluster arbitration, But as per Many MS articles Next possible node will own the cluster group. Here Node 2 is next possible owner, but why Node4 in DR(node Id4) is owning the group, why not next possible owner?..In similar way again if I stop on Node4, it should go to next possible owner (Node1), why is it going to Node2..just need to understand this concept..Thanks for all your helpAnonymous
January 31, 2015
if cluster group or DAG (PAM) is in DR site, they have some firewall issues to connect to DAG.Anonymous
November 28, 2015
You may run get-DatabaseAvailabilityGroup | set-DatabaseAvailabilityGroup