DevOps Basics: Harnessing Continuous Integration and Infrastructure As Code
Hello folks,
A little while ago, Vancouver based start-up Roomsy and Microsoft collaborated in a Hackfest to explore how Azure Web App services and DevOps best practices could address their needs. Roomsy is a cloud-based Property Management System. It provides its customer a platform to manage their rental property, take reservations that are created through a booking engines. Roomsy also provide visibility for its customer to Online Travel Agencies such as Booking.com and Expedia.
They needed to make changes to both their infrastructure and to their application code to provide a better customer and support experience. The Proof of Concept (PoC) oriented Hackfest made use of the following services and practices:
- Azure Application Services
- Azure Functions
- Azure Storage
- Infrastructure as Code
- Continuous Integration
The hack took place over the course of 20 hours with the goals of migrating Roomsy’s offering to Azure, and to enable DevOps best practices. The DevOps portion is what I really want to discuss today. If your organization has asked you to do DevOps? Or your following IT publications that are promoting the adoption of a DevOps process. And you’re not sure where to start…. Don’t worry… You’re not alone.
The term DevOps has become something of a trend. DevOps however is more than just Devs and Ops working together. It is not something you can license and deploy. It’s a culture, not a technology. It brings efficiencies that can yield increased productivity and a better understanding between IT Professionals, Developers and Business Decision Makers. Today we will look at how we addressed some of the challenges that Roomsy faced by implementing some of the DevOps practices.
DevOps’ core practices:
- Agile software development
- Continuous integration
- Continuous delivery pipelines
- Automated and continuous testing
- Proactive monitoring
- Improved communication and collaboration
Of course, you don’t need to roll ALL practices at the same time. However, they each bring a level of automation and benefits that makes the implementation worthwhile.
Roomsy operated their service in a cloud solution for hosting LAMP applications. their solution contained 3 PHP applications, and two MySQL databases located on the same server. One of the challenges we faced is that the Roomsy services, enabling management of different properties including hotels, motels, RV parking camps etc. to order and book rooms, generate reports, check-in/out guests. As such, any delays are critical because the whole business would shut down.
As a global solution, there isn’t a window of time where maintenance can be permitted to bring the system down. Therefore, a solution that minimized downtime and automated the deployment of application code and infrastructure changes was needed.
Hackfest approach
On the first day we spent some time looking at their existing development and delivery methodologies to establishing a Value Stream Map (VSM) of their existing delivery process, from conception to production.
The VSM allowed Roomsy to realize that its current way or developing and deploying their solution left them vulnerable to significant issues, with little or no ways to mitigate risks.
While Development and Staging environments did exist, they were not fully utilized efficiently in their current process.
Their existing process could be described in two steps
- Write code
- Copy code to production servers
While this seems to work today, this approach means there is no easy way to back out if a change broke the production environment or to test new features without impacting the production environment and existing customers. And no way to ensure that availability is maintained since the entire solution currently runs on a single VM.
DevOps objectives
Based on priorities and the time we had on hands, we agreed to limit ourselves to two objectives for the Hackfest. While far from ideal, the current process could not be changed in the short term, so small incremental improvements needed to be found. we agreed to work on the following topics:
1. Implement a Continuous Integration Model with the capacity to roll back any changes.
-
- As part of the Azure Migration we would identify a Continuous Integration process by which each time the Master branch of the code is merged or updated in any way. It will be automatically pushed to a dev slot in the Azure Resource Group.
2. Base the infrastructure on templates and automation to ensure that it can be replicated easily. (Infrastructure As Code)
-
- The environment in Azure will be turned into a JSON template to facilitate the capabilities of automated deployment
Solutions, steps, and delivery
Continuous Integration
Roomsy uses a private Github repository where the solution code is being stored. It provides Roomsy with an affordable way to ensure speed, data integrity, and support for distributed, non-linear workflows.
The master branch of the repo will be pushed to the Dev slot of the App service in the Roomsy Resource group. Please note that GitHub integration will be activated for Dev slot only. Once all testing activities are completed, it will be swapped with the production slots. We are not going to setup any deployment sources for production deployment for now due to the fast moving/time limited focus that a Hackfest dictates.
In the Web App blade, select Deployment Slots, and add a new slot.
Once the Dev slot is created, we used the deployment Options blade and configure GitHub as the source for this slot.
Provided the credentials, selected the branch, and saved the settings.
From that moment, whenever the Master branch of the Code Repository on GitHub is updated or a merge takes place that copy of the application code is automaticly deployed tp the dev slot of the App service. It is where it can be tested and fixed (should need be…) without any impact to the production environment. This is a great improvement over the current deployment methodology.
Infrastructure as Code
Infrastructure as code (IaC) is the process of managing and provisioning your environment through templates and definition files (in a JSON format for Azure Resource Manager Model), rather than physical hardware configuration or interactive configuration tools.
The Infrastructure as Code section was broken into two sections
- Network Configuration
- Linux/MySQL backend
The Network configuration was defined in a JSON template (the base code can be found here)
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"sites_TestBedWebApp1_name": {
"defaultValue": "TestBedWebApp1",
"type": "String"
},
"serverfarms_TestBedWebAppPlan_name": {
"defaultValue": "TestBedWebAppPlan",
"type": "String"
}
},
"variables": {
"apiVersion": "2015-06-15",
"DBVMName": "LinuxmySQL",
"DBPublicIPNSG": "DBPublicIPNSG",
"DBDNSLabel": "dbhost",
"DBPublicIPName": "[concat('DBPublicIPName', uniqueString(resourceGroup().id))]",
"DBHostNIC": "DBHostNIC",
"DBVMAdminName": "sysadmin",
"adminPublicKey": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxU2qakEIdKLScRlg3csGxrbhyj96TZ3E0bXoF78+7aJxXx2VSqUEzo+DF6reE9Ya/Y/Awvw2cSqM7420GBJeZeX8byQ6kW7Hz/8m/mXNpm7wPJpRnzy0SfAeVqZ0pW3XsCusVC/FzhY7OoL77q0pQGIbASoA8gYTYQb4uqnMm+bEuw4PoEGi0wQNXyYVzT/wW//c2aVVFuHb1kASrcQG0acOJDfx8j3oharhaxEo5sKSWxEKCHZazkUE4Gc+3MHmx162EE/YSrqjemdF9EoCpaorKFy2FyjwiC/Nmz2RVu6MXxDnHrMaLbdF6Z8LdStRJPxHRFuV4Q5/YJTwNPSVv imported-openssh-key",
"adminPassword": "P@ssw0rd!234",
"OSDiskName": "[concat(variables('DBVMName'),'osdisk')]",
"vmSize": "Standard_DS1_v2",
"gatewaySku": "Standard",
"gatewayPublicIPName": "[concat('gw1pip', uniqueString(resourceGroup().id))]",
"VNetname": "RoomsyNet",
"VNetPrefix": "10.0.0.0/16",
"VNetSubnet1Name": "Subnet-1",
"VNetSubnetPrefix": "10.0.0.0/24",
"VNetgatewayName": "Gateway",
"VNetGatewaySubnetName": "GatewaySubnet",
"VNetGatewaySubnetPrefix": "10.0.254.0/24",
"storageAccountName": "[concat(uniquestring(resourceGroup().id), 'standardsa')]",
"storageAccountType": "Standard_LRS",
"vmStorageAccountContainerName": "vhds",
"vnetRef": "[resourceId('Microsoft.Network/virtualNetworks',variables('VNetname'))]",
"SubnetRef": "[concat(variables('vnetRef'),'/subnets/',variables('VNetSubnet1Name'))]"
},
"resources": [{
"apiVersion": "2015-06-15",
"type": "Microsoft.Network/virtualNetworks",
"name": "[variables('VNetname')]",
"location": "[resourceGroup().location]",
"dependsOn": [],
"properties": {
"addressSpace": {
"addressPrefixes": [
"[variables('VNetPrefix')]"
]
},
"subnets": [{
"name": "[variables('VNetSubnet1Name')]",
"properties": {
"addressPrefix": "[variables('VNetSubnetPrefix')]"
}
},
{
"name": "[variables('VNetGatewaySubnetName')]",
"properties": {
"addressPrefix": "[variables('VNetGatewaySubnetPrefix')]"
}
}
]
}
},
{
"apiVersion": "2016-03-30",
"type": "Microsoft.Network/publicIPAddresses",
"name": "[variables('gatewayPublicIPName')]",
"location": "[resourceGroup().location]",
"comments": "This is the public IP for vNet Gateway",
"properties": {
"publicIPAllocationMethod": "Dynamic"
}
},
{
"apiVersion": "2016-03-30",
"type": "Microsoft.Network/publicIPAddresses",
"name": "[variables('DBPublicIPName')]",
"location": "[resourceGroup().location]",
"properties": {
"publicIPAllocationMethod": "Dynamic",
"dnsSettings": {
"domainNameLabel": "[variables('DBDNSLabel')]"
}
}
},
{
"type": "Microsoft.Network/networkSecurityGroups",
"name": "[variables('DBPublicIPNSG')]",
"apiVersion": "2016-03-30",
"location": "[resourceGroup().location]",
"properties": {
"securityRules": [{
"name": "SSH",
"properties": {
"protocol": "Tcp",
"sourcePortRange": "*",
"destinationPortRange": "22",
"sourceAddressPrefix": "*",
"destinationAddressPrefix": "*",
"access": "Allow",
"priority": 1000,
"direction": "Inbound"
}
},
{
"name": "MySQL",
"properties": {
"protocol": "Tcp",
"sourcePortRange": "*",
"destinationPortRange": "3306",
"sourceAddressPrefix": "*",
"destinationAddressPrefix": "*",
"access": "Allow",
"priority": 1002,
"direction": "Inbound"
}
}
]
},
"resources": [],
"dependsOn": []
},
{
"type": "Microsoft.Network/networkInterfaces",
"name": "[variables('DBHostNIC')]",
"apiVersion": "2016-03-30",
"location": "[resourceGroup().location]",
"dependsOn": [
"[resourceId('Microsoft.Network/virtualNetworks', variables('VNetname'))]",
"[resourceId('Microsoft.Network/publicIPAddresses', variables('DBPublicIPName'))]",
"[resourceId('Microsoft.Network/networkSecurityGroups', variables('DBPublicIPNSG'))]"
],
"properties": {
"ipConfigurations": [{
"name": "ipconfig1",
"properties": {
"privateIPAddress": "10.0.0.4",
"privateIPAllocationMethod": "Dynamic",
"publicIPAddress": {
"id": "[resourceId('Microsoft.Network/publicIPAddresses', variables('DBPublicIPName'))]"
},
"subnet": {
"id": "[variables('SubnetRef')]"
}
}
}],
"dnsSettings": {
"dnsServers": []
},
"enableIPForwarding": false,
"networkSecurityGroup": {
"id": "[resourceId('Microsoft.Network/networkSecurityGroups', variables('DBPublicIPNSG'))]"
}
}
},
{
"apiVersion": "2015-05-01-preview",
"type": "Microsoft.Compute/virtualMachines",
"name": "[variables('DBVMName')]",
"location": "[resourceGroup().location]",
"dependsOn": [
"[concat('Microsoft.Storage/storageAccounts/', variables('storageAccountName'))]",
"[concat('Microsoft.Network/networkInterfaces/', variables('DBHostNIC'))]"
],
"properties": {
"hardwareProfile": {
"vmSize": "[variables('vmSize')]"
},
"osProfile": {
"computerName": "[variables('DBVMName')]",
"adminUsername": "[variables('DBVMAdminName')]",
"adminPassword": "[variables('adminPassword')]",
"linuxConfiguration": {
"disablePasswordAuthentication": true,
"ssh": {
"publicKeys": [{
"path": "/home/sysadmin/.ssh/authorized_keys",
"keyData": "[variables('adminPublicKey')]"
}]
}
}
},
"storageProfile": {
"imageReference": {
"publisher": "Canonical",
"offer": "UbuntuServer",
"sku": "16.04.0-LTS",
"version": "latest"
},
"osDisk": {
"name": "osdisk",
"vhd": {
"uri": "[concat(reference(concat('Microsoft.Storage/storageAccounts/', variables('storageAccountName')), variables('apiVersion')).primaryEndpoints.blob, variables('vmStorageAccountContainerName'),'/',variables('OSDiskName'),'.vhd')]"
},
"caching": "ReadWrite",
"createOption": "FromImage"
},
"dataDisks": [{
"lun": 0,
"name": "[concat(variables('DBVMName'),'-data1')]",
"createOption": "Empty",
"vhd": {
"uri": "[concat(reference(concat('Microsoft.Storage/storageAccounts/', variables('storageAccountName')), variables('apiVersion')).primaryEndpoints.blob, variables('vmStorageAccountContainerName'),'/',concat(variables('DBVMName'),'-data1'),'.vhd')]"
},
"caching": "None",
"diskSizeGB": "1023"
}
]
},
"networkProfile": {
"networkInterfaces": [{
"id": "[resourceId('Microsoft.Network/networkInterfaces',variables('DBHostNIC'))]"
}]
}
}
},
{
"type": "Microsoft.Compute/virtualMachines/extensions",
"name": "[concat(variables('DBVMName'),'/installMySQL')]",
"apiVersion": "2015-05-01-preview",
"location": "[resourceGroup().location]",
"dependsOn": [
"[concat('Microsoft.Compute/virtualMachines/', variables('DBVMName'))]"
],
"properties": {
"publisher": "Microsoft.Azure.Extensions",
"type": "CustomScript",
"typeHandlerVersion": "2.0",
"autoUpgradeMinorVersion": true,
"settings": {
"fileUris": [
"https://raw.githubusercontent.com/pierreroman/VancHackfest/master/raid-mysql.sh"
],
"commandToExecute": "sh raid-mysql.sh"
}
}
},
{
"type": "Microsoft.Storage/storageAccounts",
"name": "[variables('storageAccountName')]",
"apiVersion": "2015-05-01-preview",
"location": "[resourceGroup().location]",
"properties": {
"accountType": "[variables('storageAccountType')]"
}
},
{
"comments": "Generalized from resource: '/subscriptions/54a522b6-6cd7-4325-b4e6-566f9d921835/resourceGroups/IACTestbed/providers/Microsoft.Web/serverfarms/TestBedWebAppPlan'.",
"type": "Microsoft.Web/serverfarms",
"sku": {
"name": "S1",
"tier": "Standard",
"size": "S1",
"family": "S",
"capacity": 1
},
"name": "[parameters('serverfarms_TestBedWebAppPlan_name')]",
"apiVersion": "2015-08-01",
"location": "Canada East",
"properties": {
"name": "[parameters('serverfarms_TestBedWebAppPlan_name')]",
"numberOfWorkers": 1
},
"resources": [],
"dependsOn": []
},
{
"comments": "Generalized from resource: '/subscriptions/54a522b6-6cd7-4325-b4e6-566f9d921835/resourceGroups/IACTestbed/providers/Microsoft.Web/sites/TestBedWebApp1'.",
"type": "Microsoft.Web/sites",
"name": "[parameters('sites_TestBedWebApp1_name')]",
"apiVersion": "2015-08-01",
"location": "Canada East",
"tags": {
"hidden-related:/subscriptions/54a522b6-6cd7-4325-b4e6-566f9d921835/resourcegroups/IACTestbed/providers/Microsoft.Web/serverfarms/TestBedWebAppPlan": "empty"
},
"properties": {
"name": "[parameters('sites_TestBedWebApp1_name')]",
"hostNames": [
"testbedwebapp1.azurewebsites.net"
],
"enabledHostNames": [
"testbedwebapp1.azurewebsites.net",
"testbedwebapp1.scm.azurewebsites.net"
],
"hostNameSslStates": [
{
"name": "[concat(parameters('sites_TestBedWebApp1_name'),'testbedwebapp1.azurewebsites.net')]",
"sslState": 0,
"thumbprint": null,
"ipBasedSslState": 0
},
{
"name": "[concat(parameters('sites_TestBedWebApp1_name'),'testbedwebapp1.scm.azurewebsites.net')]",
"sslState": 0,
"thumbprint": null,
"ipBasedSslState": 0
}
],
"serverFarmId": "[resourceId('Microsoft.Web/serverfarms', parameters('serverfarms_TestBedWebAppPlan_name'))]"
},
"resources": [],
"dependsOn": [
"[resourceId('Microsoft.Web/serverfarms', parameters('serverfarms_TestBedWebAppPlan_name'))]"
]
}
]
}
The template is needed to create the following items in the Resource group
- Virtual Network
- Virtual Network Interface card for the Linux host that will run MySQL
- Network Security Group that will look incoming ports to SSH (TCP/22) and MySQL (TCP/3306)
- A public IP address for the MySQL host with DNS name label to avoid hardcoding IP addresses
- The Application Service and the Web App where the application code will be deployed.
- The JSON template will also deploy a custom extension to the Linux machine that will configure and mount a separate Data disk where the MySQL database will store the data.
A Point-To-Site VPN Gateway to allow the Web App to connect to the backend database will be configured manually since it is not yet supported in a template.
<insert JSON Here> 
Once the machine is complete and tested. We used the information included in the following posts to capture the machine and generate a JSON file that we modified for our need. For deployment in other environment.
· Step-by-Step: Capture a linux VM Image from a running VM
· Step-by-Step: Deploy a new Linux VM from a captured image
The Value Stream Mapping was a very short and unrevealing activity considering the lack of established processes, but it helped Roomsy see the big picture and understand where automation, proper processed and DevOps practices can mitigate risks and allow for growth.
A lot of very interesting ideas on how to improve the process were discussed during this Hackfest, some more doable than others. Most importantly, though, Roomsy realized the value of continuously improving and is committed and willing to put a lot of effort into this.
General lessons
Some key points to consider:
- Automated testing needs to always be a top priority, from unit tests to integration and load tests.
- Being confident in the quality of the code is a prerequisite in order to release it.
- Automation of the deployment for the development, testing and production environments provide certitude that standards are maintained, that each environment is true to design and will lower failure risks
So, If your organization asks you to do DevOps… jump in. one automation at the time. You will see the benefits very soon.
Cheers!