Step-by-Step: Automated Provisioning for Linux in the Cloud with Microsoft Azure, XPlat CLI, JSON and Node.js ( Part 2 )

There's lots of tools that can be leveraged for automating Linux workloads on Microsoft Azure, including Azure Automation, PowerShell DSC for Linux, VM Agent Custom Scripts, Cloud-Init, XPlat CLI for Azure, Vagrant, Docker and third-party tools such as Chef and Puppet.  Azure provides a wide variety of automation options so that you can choose the tools with which you're most familiar and, in some cases, may already have an existing investment.

In Part 1 of this two-part article series, we stepped through the process for getting our Linux admin workstation setup for Azure cloud automation using the XPlat-CLI and Cloud-Init.

This article is Part 2 of this series. In this article, we'll leverage these tools for automatically provisioning an end-to-end highly available Linux server farm environment, including storage, networking, load-balancing, virtual machines and application workloads.  As we proceed through this article, we'll be build a Linux shell script that implements this provisioning logic.

Have you completed Part 1 of this series? Before stepping through this article, be sure to complete Part 1 of this two-part series to setup your Linux admin workstation for Azure cloud automation. PART 1: Step-by-Step, Automated Provisioning for Linux in the Cloud You may also be interested in these additional Linux / Open Source cloud resources: QUICK START: Building Highly Available Linux Servers in the Cloud CLOUD-TO-CLOUD: Migrating Linux VMs to Microsoft Azure from Amazon AWS DECK and RESOURCES: Open Source + Microsoft Azure PRICING: Linux Virtual Machines on Microsoft Azure

Parameters for reusability

When provisioning new Linux workloads in the cloud, there's several values that need to be specified, such as Azure subscription, datacenter region, number of VMs, and resource names, that need to be specified for the process to be successful.  To promote reusability, I'm a big fan of using input parameters in shell scripts, so we'll start our script with defining the input parameters for provisioning our environment.

#!/bin/sh

# Set variable values based on parameters passed
azureloginuname="$1" # Azure AD login ID
azureloginpass="$2" # Azure AD password
azuresubname="$3" # Azure subscription name
azureregion="$4" # Azure datacenter region
azureprefix="$5" # Unique prefix for Azure resource names
azurevmuname="$6" # Admin username for new VMsazurevmpass="$7" # Admin password for new VMs
azurevmtotal="$8" # Number of VMs to provision
azurecloudinit="$9" # Cloud-Init config file

newlinuxonazure.sh script - Input parameters

With these input parameters defined, we'll be able to use this script to provision different environments in the Azure cloud just by varying the input parameter values when calling the finished script.

Sign-in to Azure account

Before our script can perform any provisioning steps, we'll first need to sign-in to our Azure account.  When signing in, we'll need to specify an Azure Active Directory login ID that has been delegated service administrator or co-administrator access to the Azure subscription with which we'll be working.  In our script, we can sign-in with the provided Azure AD login ID and password using the azure login command from the XPlat-CLI.

# Sign-in to Azure account with Azure AD credentials

azure login --user "$azureloginuname" --password "$azureloginpass"

# Confirm that Azure account sign-in was successful
if [ "$?" != "0" ]; then
echo "Error: login to Azure account failed"
exit 1
fi

newlinuxonazure.sh script - Sign-in to Azure account

Select Azure subscription

In Azure, an account may be associated with multiple subscriptions, so we'll next need to use the azure account set command to select the specific Azure subscription in which we'll be provisioning new workloads.

# Select Azure subscription
azure account set "$azuresubname"

# Confirm that subscription is now default

azuresubdefault=$(azure account show --json "$azuresubname" | jsawk 'return this.isDefault')
if [ "$azuresubdefault" != "true" ]; then
echo "Error: Azure subscription ${azuresubdefault} not found as default subscription"
exit 1
fi

newlinuxonazure.sh script - Select Azure subscription

In the snippet above, we're confirming that the correct Azure subscription has been selected by returning output from the azure account show command in JSON format, and then piping that output to jsawk to return the isDefault value. We'll be using JSON formatted output and jsawk throughout our script to provide additional intelligence and error handling.

Define Affinity Group

In Azure, Affinity Groups help to reduce latency and improve performance between provisioned resources within an Azure datacenter region by keeping them located reasonably "close together". We can define Affinity Groups using the azure account affinity-group create command. 

# Define Azure affinity group

azureagname="${azureprefix}ag"
azure account affinity-group create --location "$azureregion" --label "$azureagname" "$azureagname"

# Confirm that Azure affinity group exists

azureagexist=$(azure account affinity-group list --json | jsawk 'return true' -q "[?name=\"$azureagname\"]")
if [ $azureagexist != [true] ]; then
echo "Error: Azure affinity group ${azureagname} not found"
exit 1
fi

newlinuxonazure.sh script - Define an Affinity Group

In the snippet above, we've provided a unique name for the new Affinity Group by leveraging the $azureprefix value that was supplied as an input parameter earlier in the script.  In addition, we've tied the Affinity Group to a particular Azure datacenter region by referencing the $azureregion value that was also provided as a script parameter.

Create Azure Storage Account

Azure Storage Accounts are used to provide blob storage for storing persistent virtual disks for Azure virtual machines. Our script can create a new storage account for storing the VM disks we'll be provisioning by calling the azure storage account create command.

# Create Azure storage account
azurestoragename="${azureprefix}stor"
azure storage account create --affinity-group "$azureagname" --label "$azurestoragename" --geoReplication "$azurestoragename"

# Confirm that Azure storage account exists

azurestorageexist=$(azure storage account list --json | jsawk 'return true' -q "[?name=\"$azurestoragename\"]")
if [ $azurestorageexist != [true] ]; then
echo "Error: Azure storage account ${azurestoragename} not found"
exit 1
fi

newlinuxonazure.sh script - Create an Azure Storage Account

In the snippet above, we're also specifying the --geoReplication option to create our Azure Storage Account with built-in asynchronous Storage Geo-Replication between Azure datacenter regions for additional disaster recovery protection.

Create Azure Virtual Network

Azure Virtual Networks provide an isolated IP address namespace in the cloud that can optionally also connect to on-premises servers and workstations via site-to-site and point-to-site VPN tunnels.  Multiple Azure virtual machines can be co-located on a common Azure virtual network to provide an internally routable IP address range that VMs can use to communicate directly with one another - just as if those VMs were running on an internal physical network.  Our script can provision a new Azure Virtual Network by using the azure network vnet create command.

# Create Azure virtual network

azurevnetname="${azureprefix}net"
azure network vnet create --affinity-group "$azureagname" "$azurevnetname"

# Confirm that Azure virtual network exists

azurevnetexist=$(azure network vnet list --json | jsawk 'return true' -q "[?name=\"$azurevnetname\"]")
if [ $azurevnetexist != [true] ]; then
echo "Error: Azure virtual network ${azurevnetname} not found"
exit 1
fi

newlinuxonazure.sh script - Create Azure Virtual Network

Note: In the snippet above, we're specifying the --affinity-group option when creating the Azure Virtual Network.  While this is currently a valid approach, Azure Virtual Networks are moving to a new Regional Virtual Network model that offers improved networking capabilities.  After the Azure XPlat-CLI is updated to support Regional Virtual Networks, you'll be able to supply the --location option as an alternative to specifying an Azure affinity group when creating a new Azure Virtual Network so that you'll be able to take advantage of these new networking capabilities.  In the meantime, if you wish to create a new Regional Virtual Network, you can certainly do so via the Azure Management Portal.

Select Linux VM Image

When provisioning our new Azure Virtual Machines, we'll need to specify the base VM image from which our new VMs will be built. This VM image can be one of the standard Linux platform images provided in the Microsoft Azure Marketplace, or it could be a custom Linux VM Image that you've created and uploaded to your Azure subscription.  Our script can select the appropriate Linux VM image by calling the azure vm image list command.

# Select Linux VM image
azurevmimage=$(azure vm image list --json | jsawk -n 'out(this.name)' -q "[?name=\"*Ubuntu-14_04_1-LTS-amd64-server*\"]" | sort | tail -n 1)

# Confirm that valid Linux VM Image is selected
if [ "$azurevmimage" = "" ]; then
echo "Error: Azure VM image not found"
exit 1
fi

newlinuxonazure.sh script - Select Linux VM Image

In the snippet above, we're using jsawk to query the list of available images based on Ubuntu 14.04 LTS, and then we select the most recent image in the returned results for use within our script.

Provision Linux VMs

We're now ready to provision our Linux virtual machines. First, we'll define a few variables that we'll use during the VM provisioning process ...

# Set variables for provisioning Linux VMs
azurednsname="${azureprefix}app" # DNS hostname for Cloud Service
azurevmsize='Small' # Azure VM instance size
azureavailabilityset="${azureprefix}as" # Availability Set name
azureendpointport='80' # Port to Load-balance
azureendpointprot='tcp' # Protocol to Load-balance
azureendpointdsr='false' # Direct Server Return, usually false
azureloadbalanceset="${azureprefix}lb" # Load-balanced Set name

newlinuxonazure.sh script - Set variables for provisioning Linux VMs

... then, in the snippet below, we'll use the azure vm create command to create each new VM.  We'll also use the azure vm endpoint command to configure load-balanced firewall endpoints that permit inbound web traffic.

# Provision Linux VMs

# Initialize Azure VM counter
azurevmcount=1

# Loop through provisioning each VM
while [ $azurevmcount -le $azurevmtotal ]
do

# Set Linux VM hostname
azurevmname="${azureprefix}app${azurevmcount}"

# Create Linux VM - if first VM, also create Azure Cloud Service
if [ $azurevmcount -eq 1 ]; then
azure vm create --vm-name "$azurevmname" --affinity-group "$azureagname" --virtual-network-name "$azurevnetname" --availability-set "$azureavailabilityset" --ssh 22 --custom-data "$azurecloudinit" --vm-size "$azurevmsize" "$azurednsname" "$azurevmimage" "$azurevmuname" "$azurevmpass"
else
azure vm create --vm-name "$azurevmname" --affinity-group "$azureagname" --virtual-network-name "$azurevnetname" --availability-set "$azureavailabilityset" --ssh $((22+($azurevmcount-1))) --custom-data "$azurecloudinit" --vm-size "$azurevmsize" --connect "$azurednsname" "$azurevmimage" "$azurevmuname" "$azurevmpass"
fi

# Confirm that VM creation was successfully submitted
if [ "$?" != "0" ]; then
echo "Error: provisioning VM ${azurevmname} failed"
exit 1
fi

# Define load-balancing for incoming web traffic on each VM
azure vm endpoint create-multiple "$azurevmname" $azureendpointport:$azureendpointport:$azureendpointprot:$azureendpointdsr:$azureloadbalanceset:$azureendpointprot:$azureendpointport

# Confirm that load-balancing config was successfully submitted
if [ "$?" != "0" ]; then
echo "Error: provisioning endpoints for VM ${azurevmname} failed"
exit 1
fi

# Wait until new VM is in a Running state
azurevmstatus='None'
while [ "$azurevmstatus" != "ReadyRole" ]
do
sleep 30s
azurevmstatus=$(azure vm show --json --dns-name "$azurednsname" "$azurevmname" | jsawk 'return this.InstanceStatus')
echo "Provisioning: ${azurevmname} status is ${azurevmstatus}"
done

# Increment VM counter for next VM
azurevmcount=$((azurevmcount+1))

done

newlinuxonazure.sh script - Provision Linux VMs

There's quite a bit occurring in this last script snippet, so let's step back and take a closer look ...

• First, we initialize an $azurevmcount variable and define a while loop that will provision each VM in turn. 
   
• Within this loop, we're setting a unique Linux hostname for each VM and calling the azure vm create command to submit a request for building a new VM with the supplied values. 
   
• When the first VM is built, the snippet will also build a new Azure Cloud Service that holds the public DNS hostname for the load-balanced VMs.
   
• For each VM that is created, they are configured in a common Azure Availability Set for placement in separate update domains and fault domains.  This configuration provides a financially-backed 99.95% Service Level Agreement (SLA) for the availability of these VMs.
   
• We call the azure vm endpoint command to configure load-balancing of incoming web traffic for each VM.
   
• After submitting provisioning requests for each VM, we check the VM status to make sure the VM has booted before continuing with provisioning the next VM.

Provision Application Workloads

As part of the VM provisioning process, our script also configures the application workloads running inside each VM. When configuring application workloads, there's several options that we can use, including Cloud-Init, Custom Scripts, Docker and PowerShell DSC for Linux, as well as third-party tools such as Chef and Puppet. In our script snippet above, we're supplying a Cloud-Init config file using the --custom-data option on the azure vm create command.

#cloud-config

# Install additional packages on first boot
#
# Default: none
#
# if packages are specified, this apt_update will be set to true
#
# packages may be supplied as a single package name or as a list
# with the format [<package>, <version>] wherein the specifc
# package version will be installed.
packages:
- apache2

cloudinit-apache.txt - Install Apache2 package inside Linux VMs

Cloud-Init provides lots of flexible options in configuring application workloads inside VMs (see the Cloud-Init documentation for more details), but in our example above we're using just a simple configuration file that ensures that the necessary package is installed for supporting web workloads in each of our Linux VMs.

Sign-out from Azure

At the end of our script, we can clean-up our admin environment and automatically sign-out of our Azure account using the azure logout command.

# End of script

echo "Success: Azure provisioning for ${azurednsname} completed"
azure logout "$azureloginuname"

newlinuxonazure.sh script - Sign-out of Azure account

Running our script

After we've saved on script file on our Linux admin workstation, we can flag the script as executable with the standard Linux chmod command ...

chmod 755 ./newlinuxonazure.sh

... and, then run our new script with our desired parameter values to test provisioning a new highly available Linux web server farm on the Azure cloud platform!

./newlinuxonazure.sh "azureadmin@contoso.com" "password" "your-azure-subscription-name" "azure-region" "unique-azure-prefix" "vm-admin-name" "vm-admin-password" total-number-of-vms "path-to-cloud-init-config-file"

After a few minutes, our script completes and our new highly available Linux web server farm is now up and running on the Azure cloud, ready for us to begin using ...

Microsoft Azure Management Portal - https://manage.windowsazure.com

.... and we can browse to our new load-balanced web service using the provisioned DNS hostname to confirm that it's available ...