Active Route Gets Removed on Windows 2008 Failover Cluster IP Address Offline
We have received calls adding static routes on Windows Server 2008 Failover Clustering nodes and wanted to pass along some important information regarding this. The issue is that when you add a static persistent route to a network adapter that is on a Windows Server 2008 Failover Cluster and take a Clustered IP Address offline (or move it to another node), the “Active” route is removed and no connections can be made using this route even though it still shows as persistent. Once you bring the Clustered IP Address back online, the active route is returned.
First, I want to mention that the networking architecture in 2008 Failover Clustering has been rewritten from the ground up and now we have our own internal route listings as well as our own adapter (Microsoft Failover Cluster Virtual Adapter). I will not into the specifics of the adapter here, but you can read more about it in the previous blog “What is a Microsoft Failover Cluster Adapter anyway?”.
On to the problem and resolution. As a little setup, here is the configuration that I want to discuss.
ClusterNode1
Physical IP Address: 10.44.60.4
Physical Subnet Mask: 255.255.0.0
Default Gateway: 10.44.60.1
ClusterNode2
Physical IP Address: 10.44.60.3
Physical Subnet Mask: 255.255.0.0
Default Gateway: 10.44.60.1
Failover Cluster Virtual IP Address
IP Address: 10.44.60.6
I also have a backup server that I use to create backups using an IP Address of 10.51.0.1 and subnet mask 255.255.0.0 that will use the same default gateway above. Most Network Administrators would use the following ROUTE.EXE command to add a persistent static route to the local tables so that a connection can be made.
C:\> route -p add 10.51.0.0 mask 255.255.0.0 10.44.60.1
So with everything online (including the Failover Cluster Virtual IP Address) on ClusterNode1, I can do a ROUTE PRINT command to display my IP Address version 4 table and see this. As a side note, I am just pulling the necessary information from the Route Table.
C:\> route print -4
IPv4 Route Table
===========================================================================
Active Routes:
Network Destination Netmask Gateway Interface Metric
10.44.0.0 255.255.0.0 On-link 10.44.60.4 276 <<—
10.44.60.4 255.255.255.255 On-link 10.44.60.4 276 <<— Node IP
10.44.60.6 255.255.255.255 On-link 10.44.60.4 276 <<— Cluster IP
10.44.255.255 255.255.255.255 On-link 10.44.60.4 276 <<—
10.51.0.0 255.255.0.0 10.44.60.1 10.44.60.4 21 <<— Route added
224.0.0.0 240.0.0.0 On-link 10.44.60.4 276 <<—
255.255.255.255 255.255.255.255 On-link 10.44.60.4 276 <<—
===========================================================================
Persistent Routes:
Network Address Netmask Gateway Address Metric
10.51.0.0 255.255.0.0 10.44.60.1 1 <<— Persistent Route added
===========================================================================
As long as the Clustered IP Address of 10.44.60.6 is online on this node, all is well. However, if I were to take the 10.44.60.6 IP Address offline, things change.
C:\> cluster res “IP Address 10.44.60.6” /offline
Taking resource ”IP Address 10.44.60.6” offline…
Resource Group Node Status
——————– ———- ————— ——
IP Address 10.44.60.6 Data Group ClusterNode1 Offline
C:\> route print -4
IPv4 Route Table
===========================================================================
Active Routes:
Network Destination Netmask Gateway Interface Metric
10.44.0.0 255.255.0.0 On-link 10.44.60.4 276 <<—
10.44.60.4 255.255.255.255 On-link 10.44.60.4 276 <<— Node IP
10.44.255.255 255.255.255.255 On-link 10.44.60.4 276 <<—
224.0.0.0 240.0.0.0 On-link 10.44.60.4 276 <<—
255.255.255.255 255.255.255.255 On-link 10.44.60.4 276 <<—
===========================================================================
Persistent Routes:
Network Address Netmask Gateway Address Metric
10.51.0.0 255.255.0.0 10.44.60.1 1 <<— Persistent Route added
===========================================================================
Notice here that the Clustered IP Address 10.44.60.6 as well as the 10.51.0.1 “Active” route is removed. Because the 10.51.0.0 route is removed, connectivity to the backup server is lost. If you bring the Clustered IP Address 10.44.60.6 online again, the “Active” routes are re–populated again and connectivity to the backup server is restored.
C:\> cluster res “IP Address 10.44.60.6” /online
Bringing resource ”IP Address 10.44.60.6” online…
Resource Group Node Status
——————– ———- ————— ——
IP Address 10.44.60.6 Data Group ClusterNode1 Online
C:\> route print -4
IPv4 Route Table
===========================================================================
Active Routes:
Network Destination Netmask Gateway Interface Metric
10.44.0.0 255.255.0.0 On-link 10.44.60.4 276 <<—
10.44.60.4 255.255.255.255 On-link 10.44.60.4 276 <<— Node IP
10.44.60.6 255.255.255.255 On-link 10.44.60.4 276 <<— Cluster IP
10.44.255.255 255.255.255.255 On-link 10.44.60.4 276 <<—
10.51.0.0 255.255.0.0 10.44.60.1 10.44.60.4 21 <<— Route Added
224.0.0.0 240.0.0.0 On-link 10.44.60.4 276 <<—
255.255.255.255 255.255.255.255 On-link 10.44.60.4 276 <<—
===========================================================================
Persistent Routes:
Network Address Netmask Gateway Address Metric
10.51.0.0 255.255.0.0 10.44.60.1 1 <<— Persistent Route added
===========================================================================
According to our Networking Development Groups, the recommendation actually is that on-link routes should be added with a 0.0.0.0 entry for the next hop, not with the local address (particularly because the local address might be deleted) and with the interface specified.
The ROUTE.EXE command has additional parameters of METRIC and INTERFACE that you would need to specify that will bind the route to the card itself.
C:\> route /?
Manipulates network routing tables.
ROUTE [-f] [-p] [command [destination]
[MASK netmask] [gateway] [METRIC metric] [IF interface]
interface the interface number for the specified route.
METRIC specifies the metric, ie. cost for the destination.
So what you need to do first is determine what the interface is so that we can bind the route to it. When doing the ROUTE PRINT or NETSH command, it will give you the interfaces at the top first. Something similar to this:
C:\> route print
IPv4 Route Table
===========================================================================
Interface List
23 …00 15 5d 4a ac 06 …… Local Gigabit Controller
19 …00 15 5d 4a ac 01 …… Local Gigabit Controller #2
18 …00 15 5d 4a ac 00 …… Local Gigabit Controller #3
===========================================================================
-or-
C:\> netsh int ipv4 show int
Idx Met MTU State Name
— — —– ———– ——————-
18 50 4294967295 connected Local Gigabit Controller #3
19 5 1500 connected Local Gigabit Controller #2
23 5 1500 connected Local Gigabit Controller
I can go into the Network and Sharing Center if I have to see which card is on this network. In my particular case, the “Local Gigabit Controller #3” is the one I want to use. So to get my persistent route to stay even though the Clustered IP Address goes offline, my command would be below. Please note that the METRIC is not needed as a requirement of the command.
C:\> route -p add 10.51.0.0 mask 255.255.0.0 0.0.0.0 metric 276 if 18
Now, the “Active” route will stay and you will have your connectivity regardless if a Clustered IP Address is online or offline. A good rule of thumb, moving forward, would be that if you are adding a static persistent route, start specifying the 0.0.0.0 and the interface as this is the proper supported commands from a networking perspective. This will result in the proper functioning no matter if Failover Clustering is configured or not.