Jaa

Diskless servers can boot and run from the Microsoft iSCSI Software Target using a regular network card!

  • Artikkeli

The new Microsoft iSCSI Software Target 3.3 includes support for a new feature called differencing virtual hard disks (VHDs). This feature helps deploy diskless boot for servers running Windows, especially in a Windows HPC Server 2008 R2 compute cluster environment.

We worked closely with the HPC team to deliver a simple management experience. The deployment process is tightly integrated within the Microsoft HPC management pack, which manages the iSCSI target using an HPC provider. Support for differencing VHDs and iSCSI boot in iSCSI Software Target 3.3 is useful in other deployments beyond HPC and this post will focus on how to deploy iSCSI boot outside of the HPC environment. You can get more details about how this works in this blog post.

Advantages of using diskless boot include:

  1. Fast deployment: A single golden image can be used to deploy many machines in parallel. At Microsoft we tested deployments as large as 256 nodes in 30 minutes.
  2. Quick recovery: Since the operating system image is not on the server, you could simply replace the blade and point to the old remote VHD, and boot from it. No operating system installation is required.
  3. Cost reduction: Because many servers can boot off of a single image, CAPEX and OPEX are directly reduced by having lower storage requirements for the operating system images. This also reduces the power, cooling, and space required for the storage.

This post will explain how diskless boot works and how you can try it out yourself!


Terminology

Before going any further, let’s clarify a few terms:

Client: The server that will boot from the image stored on the iSCSI target server. It can also be referred as an iSCSI initiator or a “diskless node”. Note: Since iSCSI boot is only supported on Windows Server SKU’s, the term “Client” refers to the iSCSI client machine which runs Windows Server OS.

Golden image: A generalized (sysprep’d) image containing an operating system and any other required applications. It is prepared on an iSCSI LUN and set to be a read-only master image. A section below describes one way to create the golden image.

HPC: High-performance computing (see here).

iSCSI target: The end point where the iSCSI initiator will establish the connection. Once the initiator connects to the iSCSI target, all the VHDs associated with that target will be accessible to that initiator.

iSCSI Software Target: Software application which provides iSCSI storage to clients (iSCSI initiators).

Differencing VHD: One of the supported VHD formats used by Microsoft iSCSI Software Target. (For a definition, please see here.) When using diskless boot, the clients will read from the golden image and write to a differencing VHD. This allows multiple clients to boot off of the same golden image.

Boot loader: Refers to the software that can bootstrap the operating system. In the iSCSI boot scenario, it contains a basic iSCSI initiator that can connect to the iSCSI target and mount a disk. It is an alternative solution to an iSCSI boot-capable NIC/HBA.

Overview

There are two phases when managing diskless clients:

  1. Deployment: Enabling a diskless physical server (bare metal) to boot Windows, involves:
    • Preparing a golden image. If you are planning to reuse the image, you can store it in a folder, and copy it out later when you do the deployment.
    • Create a one or more differencing VHDs, and set the golden image as the parent VHD
    • Create an iSCSI target for each differencing VHD and assign the target to the client.
    • Power on the client.
    • Final operating system customization – This is the Windows setup process. It can be automated using an unattended setup file.
  2. Boot process: This is the normal machine boot process, that is handled by the iSCSI boot capable hardware or software. (For diagrams on how the boot process works, see the boot process sections below).

See this TechNet article on iSCSI Boot which covers the more information on this topic. It includes a step-by-step guide for deployment using options 1 & 2 mentioned below.

Hardware/Software options

You will need one of the following options to enable iSCSI boot on a physical client:

  1. An iSCSI boot-capable NIC or
  2. An iSCSI boot-capable HBA or
  3. A software-based iSCSI boot loader



An iSCSI boot-capable NIC

The NIC needs to be configured in the server BIOS to provide the iSCSI target IQN, iSCSI target IP address, and credentials if set (examples of this type include Intel or Broadcom).

image

Figure 1 - iSCSI Boot capable NIC configuration
An iSCSI boot-capable HBA

 

An iSCSI boot capable HBA requires similar information to the iSCSI NIC. The follow example shows a QLogic HBA.

image
Figure 2 - iSCSI boot capable HBA configuration

An iSCSI boot loader

A software boot loader needs more configuration and details are in the workflow section of this post. Examples of a boot loader are gPXE (open source) and Doubletake Netboot/I (commercial). This post will use the gPXE as an example. You can also use iPXE instead of gPXE.

Boot Process

There are two stages during machine boot up:

  1. Pre-boot
  2. Windows boot

The pre-boot phase can be executed by any of the three options described above (NIC, HBA, or software boot loader).

  • For hardware, the boot loader is built in the firmware of the NICs or HBAs, and it can connect directly to the iSCSI Target and mounted to the assigned disk containing the Windows operating system image.
  • With a software boot loader, you can put the boot loader image on a CD or a USB drive, and have the computer boot to the device. Once the computer logs on to the iSCSI target, it enters the Windows boot phase.

The following diagram describes the components involved on the client computer during the boot process:

image

Figure 3 - Boot process components

When the pre-boot loader starts, it loads a real-mode network stack driver. The loader contains an iSCSI initiator which logs on to the iSCSI target and mount the boot disk to the system. Another function of the loader is to populate the iSCSI Boot Firmware Table (iBFT), which is required for iSCSI boot. The Boot Parameter driver in Windows will load the parameters from the iBFT, and the Microsoft iSCSI Software Initiator will be able to connect to the iSCSI target using the parameters set in the iBFT. The importance of the iBFT is to be able to share the parameters between the iSCSI boot initiator (which establishes the session in the preboot phase) and the Microsoft iSCSI initiator (which establish the session after Windows boots).

Various deployment configurations

If you are using hardware NICs or HBAs, please visit this TechNet page for a different approach for deployment.

Once you have a hardware or software solution selected, you need to think about whether you want to leverage DHCP and TFTP server for the deployment:

Boot loader

Using DHCP for deployment

Hardware (NIC or HBA)

If yes, see your vendor’s documentation

If not, see Using an iSCSI boot capable NIC or HBA

Software loader (gPXE)

If yes, see Use a Software boot loader with DHCP and TFTP server

If not, see Using software boot loaders without DHCP and TFTP server

 

Using an iSCSI boot-capable NIC or HBA

Using a hardware solution is easy to deploy, you just configure the firmware with the iSCSI target IQN and IP address (see Figure 1 above), then power up the client. The basic iSCSI initiator built into the firmware will be able to connect to the target and mount the VHDs.

Since it doesn’t use a software boot loader, it doesn’t need additional infrastructure such as TFTP server in the deployment. The downside is that you need to acquire the hardware, and configure each one manually. To find out how to automate the configuration, you will need to contact the hardware vendor.

Required hardware:

  • Client machine with iSCSI boot-capable NIC or HBA
  • Server running the iSCSI Software Target
The boot process

Once the client NIC has been configured, the machine will boot as illustrated below:

image
Figure 4 - Boot process using iSCSI Boot capable hardware

  1. When the client machine boots up, it reads the Target IP and IQN information, and the iSCSI NIC/HBA connects to the iSCSI target.
  2. The iSCSI Target accepts the connection, and presents a VHD as a disk to the client. The disk is then mounted on the client.
  3. The boot process proceeds as if the image resides on a local disk. Once Windows starts up, it starts the Microsoft iSCSI initiator which uses the parameters specified in the iBFT table to connect to the target. You can also find more about iBFT here.


Using a Software boot loader with DHCP and TFTP servers

A TFTP server hosts the boot loader in a central location. You can configure a DHCP server to provide each client machine a unique target. When a client boots up, it acquires the IP, TFTP server location, as well as information about the target to connect to from the DHCP server. Using DHCP in combination with the TFTP server will reduce the management operations. This is the approach used in Windows HPC diskless cluster deployments. The HPC management pack guides users to configure the DHCP and TFTP servers in a simplified manner. This configuration can scale higher with automation support. The hardest part is to configure the DHCP server. The configuration needs to adhere to the software loader specification. To get more information on how to configure the DHCP server, please visit here.

Required hardware:

  • Client machine with a NIC that supports PXE boot
  • Server with the iSCSI Software Target installed
  • Server with the DHCP and Windows Deployment Services roles installed (WDS contains the TFTP server feature)

The boot process

Once the configuration of the DHCP server is done, and the boot loader has been copied to the TFTP server, the machine boot sequence follows this process:

image

Figure 5 - Boot process when using DHCP, TFTP server with software boot loader

  1. The client machine is powered up with PXE boot enabled. It requests the IP address, the TFTP server location, as well as iSCSI target connection information, from the DHCP server. (Note, most computers today support PXE boot.)
  2. The DHCP server responds with all the information.
  3. The client machine contacts the TFTP service for the boot loader.
  4. The boot loader is copied to the client machine which has a basic iSCSI boot initiator.
  5. The client machine uses the basic iSCSI boot initiator to log on to the iSCSI target. Once the connection is established, the disk is mounted on the client machine.
  6. The boot process proceeds as if the boot image is on a local disk. Once Windows starts up, it loads the Microsoft iSCSI initiator which uses the parameters specified in the iBFT table to connect to the target.

Using a Software Boot Loader without DHCP and TFTP

In a test environment, it may not be feasible to configure a DHCP and TFTP server. Without a TFTP server, you need to find another way to get the boot loader image presented to the client machines. You can put the boot loader on a USB drive or a CD in the client machine. This is the simplest way to try out and does not require any extra hardware investment. The step-by-step guide below focuses on this configuration.

Required hardware:

  • A client machine with a NIC.
  • A device (CD/USB, etc.) that can store the software boot loader
  • A server with the iSCSI Software Target installed
The boot process

Once the software boot loader has been configured, the machine boot will follow this process:

image
Figure 6 – Boot process for local software boot loader

  1. Set the client BIOS boot order to use the USB/or CD where the boot loader is stored.
  2. When the client machine boots, the boot loader uses the iSCSI target IP and IQN information, and connects to the iSCSI target.
  3. The iSCSI target accepts the connection and presents the VHD to the client. At this point the disk will be mounted on the client.
  4. The boot process proceeds as if the boot image resides on a local disk. Once Windows starts up, it loads the Microsoft iSCSI initiator. The iSCSI initiator will use the parameters specified in the iBFT table to connect to the target.

Step-by-step deployment guide

Here is the recipe for using a software boot loader on a CD/USB.

There are two stages:

  1. Create a golden image
  2. Deploy diskless clients using a golden image

After stage 2, the client machine will be ready to run a workload. You can use the machine just like one that has an OS image stored locally.

A word of caution: While the iSCSI target is servicing diskless clients, rebooting the iSCSI target server will cause unexpected behavior for the diskless clients. This is similar to removing the local hard drive while the machine is running. If you require high availability in your environment, you should consider deploying a failover cluster with the iSCSI targets LUNs.

Create a Golden Image

To create a golden image, you need to do a normal OS and application install on the iSCSI VHD, to ensure the setup process can load the driver stacks in the correct order. This is important, because the Windows setup process goes through different path for installing the OS on local hard drive vs on a iSCSI disk. The image installed on a local hard drive can not be used for iSCSI boot, as the drives loading sequences will be different.

Once the installation is complete, run sysprep on the OS image.

Steps to create a golden image:

On the iSCSI target server:

  • Create a 30GB fixed VHD (big enough to install the OS and applications).
  • Create an iSCSI target.
  • Assign the VHD to the target.
  • Assign the target to the client iSCSI IQN.

On the client machine:

  • Configure the boot loader:

  • Set the boot order to boot from gPXE media.

  • Set the system BIOS boot order to boot from OS installation media second.

  • Boot the disk client.

  • gPXE will fail because there is no OS to load, but the iSCSI Connection will be established. (See the scripts below for an explanation.)

  • Choose the OS installation media to boot (this leads to a normal OS setup).

  • Select the iSCSI LUN to install the OS and finish OS installation normally.

  • Now you have an OS installation on an iSCSI LUN.

  • Optional: Install additional applications which you want in the sysprep image.

  • Sysprep the image of the first client machine. (Find more details on sysprep here.)

  • Use the syspreped image as the parent VHD for future deployment

(Note, I used this page as reference when I did my setup. You may find it useful as well.)

Custom Script

dhcp net0
set initiator-iqn iqn.1991-05.com.microsoft:iscsiboot-${net0/mac}
set root-path iscsi:10.121.28.150::::iqn.1991-05.com.microsoft:testsvr -iscsiboot-${net0/mac}-target
set keep-san 1
sanboot ${root-path}

How to customize the boot loader script

The above script needs to be customized to your configuration:

  • Line 1: Assume booting from first NIC
  • Line 3:
    • Replace “10.121.28.150” with the actual iSCSI Target server address.
    • Replace “testsvr” with the actual iSCSI Target server host name.
    • Use IPv4, since the current version of the gPXE doesn’t support IPv6.
    • gPXE will use the standard iSCSI TCP port 3260 and LUN 0.
  • Line 4 : The command “set keep-san 1” means keep iSCSI connection when failure occurs. By setting this, you will be able to install the OS image onto the iSCSI LUN when creating the golden image.

Deploy diskless clients using a Golden image

Once the golden image is prepared, you can use it to deploy more clients by following the steps below:

  • On the target server:
    • Create a differencing VHD using WMI scripts (see this sample), and specify the base VHD which contains the golden image which is the sysprep’d OS images created from the above section.
    • Create an iSCSI target, and assign it to the client.
    • Assign the differencing VHD to the target.
  • On the client machine:
    • Boot the client machine. Because the base OS contains the image, it will load Windows, and continue with the OS finalization phase, where all of the unique information is saved to the differencing VHD.
    • P.S. Now you are done, enjoy your new diskless servers!

Jian Yan
Program Manager
iSCSI Software Target

Comments

  • Anonymous
    January 01, 2003
    @ Josh, Thanks the additional information!

  • Anonymous
    January 01, 2003
    @Hemant, send you email offline @syarif,

  1. If you use static IP, the boot process is described in Using a Software Boot Loader without DHCP and TFTP section of the article. Not sure if that answers your question.
  2. Do you mean Thumbdive for storing the ipxe file? yes, it should work.
  3. By assign VHD to a particular target, and only specific client can logon to that target, and therefore seeing the storage only available to that client.

  • Anonymous
    January 01, 2003
    @MSFT tester...many months later, but if there is no tool to do so, pretty sure you could use windows startup recovery to fix the boot path issue-use the WAIK (windows automated installation kit), or whatever they're calling the new version, to make the gold image from your traditional hdd based installation, then use startup recovery to build the new boot path...messy and time consuming, but should work

  • Anonymous
    May 12, 2011
    It worked for me, though I fumbled through it for awhile.  Definitely check out Differencing VHDs. Some stuff that was not abundantly clear to me at the outset: The gPXE script goes in the "embedded script" textarea here: rom-o-matic.net/.../build.php (I selected one of the "PXE bootstrap loader" image options). If you've never worked with iSCSI targets, here is a simplified gPXE script that should be clearer (note the given example has a space before "-iscsiboot" in the target name that threw me off): dhcp net0 set initiator-iqn testIQN set root-path iscsi:192.168.1.100::::iqn.1991-05.com.microsoft:jc-server-testvm-target set keep-san 1 sanboot ${root-path} The important parts are the "testIQN", which is whatever you set your iSCSI target to expect as an incoming initiator name (IQN); and everything after "::::", which you can get by looking at the properties of your iSCSI target (the "IQN" field). For home users that have flashed their router with some DD-WRT flavor, if you're using DNSMasq, put the following in the "Additional DNSMasq options" textarea: dhcp-boot=gpxe.pxe,,192.168.1.100 (the "gpxe.pxe" is the arbitrary name you give to the file you get from the rom-o-matic site; and the IP address is the IP address to your TFTP server)

  • Anonymous
    June 10, 2011
    it works with Algostor NAS server WSS2008R2 and Windows 7... just have to do not forget to disable the standby if not after 20 min the OS crash... www.serveur-nas.fr/.../exemple-de-boot-sur-iscsi-target-windows-storage-server-2008r-a-partir-dune-workstation-sans-disque-dur in french language

  • Anonymous
    September 29, 2011
    How would you set it up if the client workstations all had different motherboards and likely needed different kinds of "Golden images" for each? I'm chainloading PXE->gPXE->iSCSI-specific-target-per-motherboard and have about 3 or 4 different kinds of workstations on this network. How on the server side could you specify which workstation gets which target. And to make it more complicated, I can't burn a gPXE rom into these NICs as they're built into the motherboard. Otherwise I would skip chainloading altogether if I could.

  • Anonymous
    April 13, 2012
    I owns a school. School is under construction currently. I am planning to buy some computers for computer lab in the school. will it be feasible to establish the 'diskless node' environment in the computer lab or should i go for normal personal computers only? Please reply ASAP. My Email ID is hemantmangal2000@gmail.com Thanks in advance!!!

  • Anonymous
    April 15, 2012
    Regarding the diskless node. I have a doubt regarding:

  1. How to set static IP to the client (so im able to recognize who owns the IP).
  2. Thumbdrive can work or not?
  3. How to set a storage for each client? Thats mean, if IP 10.0.4.34, then it will whow his storage only.

  • Anonymous
    February 28, 2013
    Not easy to treat with many configurations like IP address and computer names. by comparing with intergrated software like: sandeploy boot server from http:/www.sandeploy.com/

  • Anonymous
    June 19, 2013
    "The image installed on a local hard drive can not be used for iSCSI boot, as the drives loading sequences will be different." - Do we know what it takes to convert an image from local disk to an iscsi bootable image ?

  • Anonymous
    September 10, 2014
    Hi, I have configured clients and server such that disk less clients boot over the network. I have installed Win7 on LUNs, the problem I am facing is, when client boots into Windows 7 over the network and if no operation is performed within next 45-60 mins, then Win7 freezes on the client, in this case I forcefully restart the client. I want to overcome the Freezing issue and also overcome the manual restart of client issue, so that my automation can be 100% complete.

    I have 4 clients and 1 server, is the client OS freezing due to multiple clients requesting I/O over network? I am not sure so please give my any idea you get, anything.

    You can contact me at: nilesh.mali90@hotmail.com

    Thanks,
    Nilesh

  • Anonymous
    February 10, 2015
    try jentu - www.jentu.ca

  • Anonymous
    April 24, 2015
    It worked for me... good one...thanks for sharing

  • Anonymous
    October 10, 2015
    Does the windows desktop editions, like windows 8.1 start the iscsi initiator by default when installed to an iscsi target?
    Thought this is only done for server os.

  • Anonymous
    February 21, 2016
    FWIW, the keep-san option is essential. If you don't read the instructions carefully (like me) and overlook this command, then Windows Setup will give the error "We couldn't create a new partition or locate an existing one" when you try to select the disk to install to.

    Thanks for the clear instruction guide.

  • Anonymous
    March 11, 2016
    Hi,

    I followed your directions and those of Bjorn (http://jentofts.net/sysadmwiki/index.php?title=Diskless_Windows_clients_using_iSCSI) to setup my iSCSI target and TFTP server. My server is running Windows Server 2008 R2 and I am trying to boot Windows 7 Embedded. As the directions say, on initial boot it failed because the VHD on the iscsi target is empty. iPXE told me that my drive was mapped to 0x80 and that it was preserved after the failed boot. When I try to run the install CD (2nd boot option) I see the VHD on the iscsi target but Windows won't let me install to it. It says that "Windows cannot be installed to this disk. This computer’s hardware may not support booting to this disk. Ensure that the disk’s controller is enabled in the computer’s BIOS menu." Am I doing something wrong? I have tried partitioning the VHD and that did not help. Any help is appreciated.

    Thanks!

  • Anonymous
    March 14, 2016
    I figured out my issue. I needed to download ipxe.kpxe and not undionly.kpxe.