Deploying Infrastructure on Azure with Azure Resource Manager & Python SDK.
Once we have gone through how to authenticate the application with Azure AD through the multi-tenant approach and single tenant approach, in this blog we will look at deploying a VM on a VNet, linked to a storage account. Typically when you are looking at automating workloads, you would like to simplify user input as much as possible.
This application loads a simple form @ https://127.0.0.1:8000/deploy that takes inputs from the user. This is a simplified sample of course, but in the back-end it could be either creating an ARM template and deploying, which is a preferred route for large scale deployments or deploy single components at a time for small scale/adhoc requirements.
We will look at the latter today, and in the coming few days we could look at the template based deployment.
You can download and install the latest version of Azure Python SDK with the pip installer:
pip install azure
This is the link to view the SDK in detail.
https://github.com/Azure/azure-sdk-for-python
Coming back to our application, following is the loading screen, assuming that the user has logged in with their application user name & password.
Figure 1: Loading screen
If we take a look at the deployment option module, views.py. The def deployment method loads the admin settings (which is the client_id & secret of the application registered with the AD it in subscription it is hosted in).
When the screen is loaded, we pre-load it with list of regions available today, storage account types and the various VM sizes available. The code has a hard coded else part, which could be replaced by reading from local DB settings.
To call any Python SDK function, we need to send two parameters for all the functsions:
1. Access token
2. Subscription IDs
Since this simply retrieving a standard list we will use the application’s own client_id and secret using the following method.
We first extract it from the default Django storage:
if AuthSettings_Admin.objects.filter(user_id="Admin").exists():
admin_authsettings = AuthSettings_Admin.objects.filter(user_id="Admin")
for admin_auth in admin_authsettings:
client_id = admin_auth.client_id
tenant_id=admin_auth.tenant_id
subscription_id=admin_auth.subscription_id
client_secret =admin_auth.client_secret
Using these we get the access token generated calling the method:
token = get_access_token(tenant_id=tenant_id,client_id=client_id,client_secret=client_secret)
The method get_access_token described below is based on raw REST call, we now have an inbuilt ADL library for Python just released, and will update the article to add that as well in some time:
Make sure you urlencode the client secret in the body.
def get_access_token(tenant_id,client_id,client_secret):
url = "https://login.microsoftonline.com/" + tenant_id + "/oauth2/token"
body_data = "&grant_type=client_credentials&resource=https://management.core.windows.net/&
client_id="+ client_id + "&client_secret="+ urllib.quote_plus(client_secret)
#body_data = urllib.urlencode(body_data_json)
headers = {"Content-Type":"application/x-www-form-urlencoded"}
req = Request(method="POST",url=url,data=body_data)
req_prepped = req.prepare()
s = Session()
res = Response()
res = s.send(req_prepped)
access_token_det = {}
if (res.status_code == 200):
responseJSON = json.loads(res.content)
access_token_det["details"]= responseJSON["access_token"]
access_token_det["status"]="1"
access_token_det["exp_time"]=responseJSON["expires_in"]
access_token_det["exp_date"]=responseJSON["expires_on"]
access_token_det["accessDetails"]=responseJSON
else:
access_token_det["details"]= str(res.status_code) + str(res.json())
access_token_det["status"]="0"
return access_token_det
Once you get the access token, we call the following functions of the SDK to get the list of regions, vm sizes and storage account types:
We have added a layer of abstraction between the SDK functions and the Django application. This is to avoid changing the application if the function’s return types change.
Please refer to the module arm_framework on this link.
loc_list = arm_fw.get_list_regions(token["details"],subscription_id)
pricing_tier = arm_fw.get_list_vm_sizes(token["details"],subscription_id)
stor_acc_types = arm_fw.get_list_storage_types(token["details"],subscription_id)
We send all of these as template parameters to the html page. Please look at the template file for details on how it is loaded, it’s typical Django & and javascript code.
Now that we have the form loaded, the user can enter the deployment name and everything gets populated.
Next step is to Get VM Image list. The Azure Resource Manager provides VM list based on location, and in multi-Step:
1. Get Publisher
2. Get Offer for the publisher
3. Get SKUs for the offer, publisher
4. Get Versions for the offer, publisher and SKUs
The form first loads the list of publisher, it is loading as a raw list which is not a very good way of loading, ideally one must provide a search box and return search results based on that.
We use the following function from SDK to load the list of publishers (getvmimagepublishers):
pub_list = arm_fw.get_list_vm_publishers(access_token=acc_token, subscription_id=subscription_id,region_name=region_name)
Based on the publishers, we call the rest of the functions:
Offers:
offer_list = arm_fw.get_list_vm_offers(access_token=acc_token, subscription_id=subscription_id,region_name=region_name,publisher_name=publisher_name)
SKUs:
skus_list = arm_fw.get_list_vm_skus(access_token=acc_token, subscription_id=subscription_id,region_name=region_name,publisher_name=publisher_name,
offer_name=offer)
Versions:
versions_list = arm_fw.get_list_vm_versions(access_token=acc_token, subscription_id=subscription_id,region_name=region_name,publisher_name=publisher_name,
offer_name=offer,sku_name=skus)
Note: You will notice, that for some publishers the version list returns empty. That means that VM image does not exist. While this application doesn’t handle it, you must ideally cater to it by providing appropriate messages.
Fetch Subscriptions:
The last bit is to fetch the list of subscriptions that the user initially had entered into settings manually or through the OpenID connect methodology explained earlier. This is done through list stored in the local DB.
Submit Deployment
This is the last step of the deployment method. First thing the client determines is whether it is an app-only authentication or user-based authentication. This it does by calling the web method on Django server as shown below in the html template file:
If it’s app-only it is very simple, the client sends all the deployment details in a JSON object directly to the server:
url: '../authdetails/',
type: 'POST',
data: deploymentDetails,
dataType: 'json',
headers: { "X-CSRFToken": getCookie('csrftoken') },
error: function(e){console.log(e.message)},
success: function (data) {
deploymentDetails["auth_type"] = data["auth_type"]
//console.log(data["auth_type"])
if (data["auth_type"] == "A") {
$.ajax({
url: '../deploy/',
type: 'post',
headers: { "X-CSRFToken": getCookie('csrftoken') },
data: deploymentDetails,
dataType: 'json',
success: function (data) {
$("#deploy_results").html(JSON.stringify(data))
$("#dep_spinner").hide()
}
})
However, if it’s the user-based authentication, the client opens a pop-up with the auth_url that the server returns:
var url = data["auth_code_url"]
popup = window.open(url, "Login", width=300,height=200);
This is to get the authorization code that will be used to generate the token.
The views.py constructs the auth_code_url using the client_if & tenant_id saved by the user as shown below.
authorize_url = "https://login.microsoftonline.com/" + tenant_id +"/oauth2/authorize?client_id="+client_id+"&response_type=code"
auth_response["auth_type"] = auth_type
auth_response["auth_code_url"] = authorize_url
auth_response["auth_token"]= None
The popup window loads the user Azure AD login page. Once the user successfully logs in, it returns the control back to the Django Server as provided in the redirect URI and passes the access code as a query string.
This function immediately returns the control back to the template calling it’s get_Access_code function.
def get_access_code(request):
query_string = request.META["QUERY_STRING"]
code_det = {"code":query_string}
htmlstring = "<html><head></title><script type='text/javascript'>window.opener.get_access_code("+json.dumps(code_det)+");window.close();</script></head><body></body></html>"
return HttpResponse(htmlstring)
The javascript function get_access_code is called as described below:
code_str = code["code"]
var code_arr = ""
code_arr = code_str.toString().split('&')
code_val = code_arr[0].replace("code=", "")
deploymentDetails["auth_code"] = code_val
$.ajax({
url: '../deploy/',
type: 'post',
headers: { "X-CSRFToken": getCookie('csrftoken') },
data:deploymentDetails,
dataType:'json',
success: function (data) {
$("#dep_spinner").hide()
$("#deploy_results").html(JSON.stringify(data))
}
The function sends the details to the same Django application function create_deployment() as described below:
The method gets the access token either directly through the app-only route or using the authorization code:
if auth_type == "U":
auth_code = request.POST["auth_code"]
token_url = "https://login.microsoftonline.com/" + tenant_id+"/oauth2/token"
body="grant_type=authorization_code&code="+auth_code+"&redirect_uri="+redirect_uri+"
&client_id="+client_id+"&client_secret="+ urllib.quote_plus(client_secret)+"&resource=https://management.core.windows.net/"
headers ={"Content-Type":"application/x-www-form-urlencoded"}
req = Request(method="POST",url=token_url,data=body)
req_prepped = req.prepare()
s = Session()
res = Response()
res = s.send(req_prepped)
responseJSON = json.loads(res.content)
token = responseJSON["access_token"]
token_result = {"POSTURL":token_url,"body":body,"headers":headers,"Response":responseJSON}
else:
res = get_access_token(tenant_id=tenant_id,client_id=client_id,client_secret=client_secret)
token_result = {"Response":res}
if(res["status"]=="1"):
token = res["details"]
else:
token=None
deploymentDetails = request.POST
subscription_id=deploymentDetails["subscriptionID"]
Create Resource group:
res_grp_created_status = arm_fw.create_resource_group(access_token=token,subscription_id=subscription_id,resource_group_name=
deploymentDetails["resourceGroupName"],region=deploymentDetails["location"])
This sends the details to our abstract layer which in turn calls the Create Vnet function in the Python SDK:
def create_resource_group(access_token, subscription_id,resource_group_name, region):
cred = SubscriptionCloudCredentials(subscription_id, access_token)
resource_client = ResourceManagementClient(cred, user_agent='SDKSample/1.0')
resource_group_params = ResourceGroup(
location=region,
tags={
'RGID': subscription_id + resource_group_name,
},
)
result_create = resource_client.resource_groups.client.resource_groups.create_or_update(resource_group_name, resource_group_params)
success=False
if result_create.status_code == 200:
success = True
elif result_create.status_code == 201:
success = True
else:
success = False
result_json = {"success":success, "resource_created":result_create.resource_group.name}
return result_json
Create Virtual Network
if the resource group gets created successfully, we then create the Virtual Network
if res_grp_created:
vnet_created_status = arm_fw.create_virtual_network(access_token=token,subscription_id=subscription_id,
resource_group_name=res_grp_name_created,virtualnetwork_name=deploymentDetails["vnetName"],region=deploymentDetails["location"], subnets=None,addresses=None,dns_servers=None)
As described in the function below, it is creating a single address space, subnet and dns server
def create_virtual_network(access_token, subscription_id,resource_group_name, virtualnetwork_name, region, subnets=None,
addresses=None, dns_servers=None):
cred = SubscriptionCloudCredentials(subscription_id, access_token)
resource_client = ResourceManagementClient(cred, user_agent='SDKSample/1.0')
resource_client.providers.register('Microsoft.Network')
network_client = NetworkResourceProviderClient(cred)
network_parameters = None
if (subnets != None and addresses != None and dns_servers != None):
addr_space = AddressSpace(address_prefixes=addresses)
dhcp_opts = DhcpOptions(dns_servers=dns_servers)
subnet_det = subnets
network_parameters = VirtualNetwork(location=region, virtualnetwork_name=virtualnetwork_name,
address_space= addr_space,dhcp_options=dhcp_opts,subnets=subnet_det)
else:
if DefaultNetworkSettings.objects.filter(setting_type_id="default").exists():
def_settings = DefaultNetworkSettings.objects.filter(setting_type_id="default")
add_sp=None
sb_net_prefix=None
sb_net_name=None
for ds in def_settings:
add_sp = ds.default_address_space
sb_net_prefix = ds.default_address_range
sb_net_name = ds.default_subnet_name
addr_space = AddressSpace(address_prefixes=[add_sp])
subnet_det = [Subnet(name=sb_net_name,address_prefix=sb_net_prefix)]
network_parameters = VirtualNetwork(location=region, virtualnetwork_name=virtualnetwork_name,address_space=addr_space,dhcp_options=None,
subnets=subnet_det)
result_create=network_client.virtual_networks.create_or_update(resource_group_name,virtualnetwork_name,network_parameters)
success=False
if result_create.status_code == 200:
success = True
elif result_create.status_code == 201:
success = True
else:
success = False
result_json = {"resource_created": virtualnetwork_name, "success":success,"subnet_name":sb_net_name}
return result_json
Create Storage Account
stor_acc_created_status = arm_fw.create_storage_account(access_token=token,subscription_id=subscription_id,
resource_group_name=res_grp_name_created,storage_account_name=deploymentDetails["storageAccountName"],region=deploymentDetails["location"],storage_type=deploymentDetails["storageAccountType"])
Following is the underlying function for creating storage account:
def create_storage_account(access_token, subscription_id,resource_group_name, storage_account_name, region, storage_type):
cred = SubscriptionCloudCredentials(subscription_id, access_token)
resource_client = ResourceManagementClient(cred, user_agent='SDKSample/1.0')
resource_client.providers.register('Microsoft.Storage')
storage_client = StorageManagementClient(cred, user_agent='SDKSample/1.0')
storage_params = StorageAccountCreateParameters(
location=region,
account_type=storage_type,
)
result_create = storage_client.storage_accounts.create(resource_group_name,storage_account_name,storage_params)
success=False
if result_create.status_code == 200:
success = True
elif result_create.status_code == 201:
success = True
else:
success = False
result_json = {"success":success, "resource_created":storage_account_name}
return result_json
Create Virtual Machine
Finally we call the create Virtual Machine sending the created details.
vm_created_status = arm_fw.create_virtual_machine(access_token=token,subscription_id=subscription_id,resource_group_name=res_grp_name_
created,vm_name=deploymentDetails["vmName"],vnet_name=vnet_name_created,subnet_name=subnet_name_created,
vm_size=deploymentDetails["vmPricingTier"],storage_name=stor_acc_created_name,region=deploymentDetails["location"],vm_username=deploymentDetails["vmUserName"],vm_password=deploymentDetails["vmPassword"],publisher=deploymentDetails["vmPublisher"],offer=deploymentDetails["vmOffer"],sku=deploymentDetails["vmSKU"],version=deploymentDetails["vmVersion"])
With underlying function as:
def create_virtual_machine(access_token, subscription_id,resource_group_name, vm_name, vnet_name,
subnet_name, vm_size, storage_name, region, vm_username, vm_password,
publisher, offer, sku, version):
interface_name=vm_name + 'nic'
cred = SubscriptionCloudCredentials(subscription_id, access_token)
resource_client = ResourceManagementClient(cred)
resource_client.providers.register('Microsoft.Compute')
compute_client = ComputeManagementClient(cred)
hardware_profile = HardwareProfile(virtual_machine_size=vm_size)
os_profile = OSProfile(computer_name = vm_name, admin_username = vm_username, admin_password = vm_password)
image_reference = ImageReference(publisher=publisher, offer=offer, sku=sku, version=version)
storage_profile = StorageProfile(os_disk=OSDisk(caching=CachingTypes.none, create_option=DiskCreateOptionTypes.from_image,
name=vm_name, virtual_hard_disk=VirtualHardDisk(
uri='https://{0}.blob.core.windows.net/vhds/{1}.vhd'.format( torage_name,vm_name)
)), image_reference=image_reference )
nic_reference = create_network_interface(access_token,subscription_id,resource_group_name,interface_name, vnet_name, subnet_name, region)
nw_profile = NetworkProfile(
network_interfaces=[
NetworkInterfaceReference(reference_uri=nic_reference)
]
)
vm_parameters = VirtualMachine(location=region, name=vm_name, os_profile=os_profile, hardware_profile= hardware_profile,
network_profile=nw_profile, storage_profile=storage_profile, image_reference=image_reference)
result_create = compute_client.virtual_machines.create_or_update(resource_group_name, vm_parameters)
success=False
if result_create.status_code == 200:
success = True
elif result_create.status_code == 201:
success = True
else:
success = False
result_json = {"resource_created": vm_name, "success":success}
return result_json
The application right now returns the names of the created resources as is, one could modify this to show a user friendly message.
Refresh Token
Please note, ideally you should be adding a check to ensure the access token does not expire, and to call the refresh token before it expires.