Faster Live Migration with RDMA in Windows Server 2012 R2

Today I want to go deep on faster live migration using RDMA.  In the past I have talked about faster live migration with compression – which is the default option for live migration in Windows Server 2012 R2 – but to me, faster live migration with RDMA is the more interesting topic to discuss.

The goal of faster live migration with RDMA is to take advantage of RDMA network acceleration in order to give you the fastest live migration possible.  We actually do this by taking advantage of SMB Direct – and actually perform the live migration over the top of SMB in this environment.

Not only does this give us access to RDMA network acceleration, but it also gives us access to other SMB benefits like native multi-channel support.  This all comes together to allow us to reach amazing speeds, with practically zero CPU overhead in the process.

Unlike live migration with compression, live migration with RDMA is not affected by the workload inside the virtual machine.  The primary thing that will affect the speed of your live migration is how many connection you have, and how fast those connections are.

So what do you need to do to get this working?  Well – simply put you need an environment that supports SMB Direct.  Thankfully Jose Barreto has already done a very detailed series of blogs on how to get SMB Direct working – so you can follow the directions here:

• http://blogs.technet.com/b/josebda/archive/2012/07/31/deploying-windows-server-2012-with-smb-direct-smb-over-rdma-and-the-mellanox-connectx-2-connectx-3-using-infiniband-step-by-step.aspx
• https://blogs.technet.com/b/josebda/archive/2012/07/31/deploying-windows-server-2012-with-smb-direct-smb-over-rdma-and-the-intel-neteffect-ne020-card-using-iwarp-step-by-step.aspx
• https://blogs.technet.com/b/josebda/archive/2012/07/31/deploying-windows-server-2012-with-smb-direct-smb-over-rdma-and-the-mellanox-connectx-3-using-10gbe-40gbe-roce-step-by-step.aspx
• https://blogs.technet.com/b/josebda/archive/2012/07/31/deploying-windows-server-2012-with-smb-direct-smb-over-rdma-and-the-chelsio-t4-cards-using-iwarp-step-by-step.aspx

Once you have it all setup – you can utilize the new performance counters that we have under “Hyper-V VM Live Migration” to monitor live migration activity over SMB Direct:

Note – you can actually enable live migration over RDMA on a system that does not support RDMA.  In this case you will get live migration over SMB (not SMB direct).

Cheers,
Ben

Comments

• Anonymous
  May 14, 2014
  Assuming you have Windows Server 2012 R2 Hyper-V servers and SMB Direct is configured per Jose's blog, how do you enable RDMA for live migration?  Or is it enabled by default once SMB Direct is available? And do we want to/need to disable compression?