How to Deploy Azure Role Pre-Requisite Components from Blob Storage
In Windows Azure SDK1.3 we have introduced the concept of startup tasks that allow us to run commands to configure the role instance, install additional components and so on. However, this functionality requires that all pre-requisite components are part of the Azure solution package.
In practice this has following limitations:
- if you add or modify pre-requisite component you need to regenerate the Azure solution package
- you will have to pay the bandwidth charge of transferring the regenerated solution package (perhaps 100s of MB) even though you actually want to update just one small component
- the time to update entire role instance is incremented by the time it takes to transfer the solution package to Azure datacenter
- you cannot update individual component rather you update entire package
Below I describe an alternative approach. It is based on the idea to leverage blob storage to store the pre-requisite components. Decoupling of pre-requisite components (in most cases they have no relationship with the Azure role implementation) has a number of benefits:
- you do not need to touch Azure solution package, simply upload a new component to blob container with a tool like Windows Azure MMC
- you can update an individual component
- you only pay a bandwidth cost for component you are uploading, not the entire package
- your time to update the role instance is shorter because you are not transferring entire solution package
Here is how the solution is put together:
When Azure role starts, it downloads the components from a blob container that is defined in the .cscfg configuration file
<Setting name="DeploymentContainer" value="contoso" />
The components are downloaded to a local disk of the Azure role. Sufficient disk space is reserved for a Local Resource disk defined in the .csdef definition file of the solution – in my case I reserve 2GB of disk space
<LocalStorage name="TempLocalStore" cleanOnRoleRecycle="false" sizeInMB="2048" />
Frequently, the pre-requisite components are installers (self extracting executables or .msi files). In most cases they will use some form of temporary storage to extract the temporary files. Most installers allow you to specify the location for temporary files but in case of legacy or undocumented third party components you may not have this option. Frequently, the default location would the directory indicated by %TEMP% or %TMP% environment variables. There is a 100MB limit on the size of the TEMP target directory that is documented in Windows Azure general troubleshooting MSDN documentation.
To avoid this issue I implemented the mapping of the TEMP/TMP environment variables as indicated in this document. These variables point to the local disk we reserved above.
private void MapTempEnvVariable()
{
string customTempLocalResourcePath =
RoleEnvironment.GetLocalResource("TempLocalStore").RootPath;
Environment.SetEnvironmentVariable("TMP", customTempLocalResourcePath);
Environment.SetEnvironmentVariable("TEMP", customTempLocalResourcePath);
}
The OnStart() method of the role (in my case it is a “mixed Role” - Web Role that also implements a Run() method) starts the download procedure on a different thread and then blocks on a wait handle.
public class WebRole : RoleEntryPoint
{
private readonly EventWaitHandle statusCheckWaitHandle = new ManualResetEvent(false);
private volatile bool busy = true;
private const int ThreadPollTimeInMilliseconds = 1500;
//periodically check if the handle is signalled
private void WaitForHandle(WaitHandle handle)
{
while (!handle.WaitOne(ThreadPollTimeInMilliseconds))
{
Trace.WriteLine("Waiting to complete configuration for this role .");
}
}
[...]
public override bool OnStart()
{
//start another thread that will carry would configuration
var startThread = new Thread(OnStartInternal);
startThread.Start();
[...]
//Blocks this thread waiting on a handle
WaitForHandle(statusCheckWaitHandle);
return base.OnStart();
}
}
If you block in OnStart() or continue depends on the nature of the pre-requisite components and where they are used. If you need your components to be installed before you enter Run() – you should block because Azure will call this method pretty much immediately after you exit OnStart(). On the other hand if you have a pure Web role than you could choose not to block. Instead, you would handle the StatusCheck event and indicate that you are still busy configuring they role – the role will be in Busy state and will be taken out of the NLB rotation.
private void RoleEnvironmentStatusCheck(object sender, RoleInstanceStatusCheckEventArgs e){
if (this.busy)
{
e.SetBusy();
Trace.WriteLine("Role status check-telling Azure this role is BUSY configuring, traffic will not be routed to it.");
}
// otherwise role is READY
return;
}
If you choose to block OnStart() there is little point to implement the StatusCheck event handler because when OnStart() is entered the role is in Busy state and the status check event is not raised until OnStart() exits and role moves to Ready state.
The function OnStartInternal() is where the components are downloaded from the preconfigured Blob container and written to local disk that we allocated above. When downloading the files we look for a file named command.bat.
This command batch file holds commands that we will run after the download completes in the similar way as we do it with startup tasks. Here the sample command.bat
vcredist_x64.exe /q
exit /b 0
I should point out that if you are dealing with large files you should definitely specify the Timeout property in the BlobRequestOptions for the DownloadToFile() function. The timeout is calculated as a product of the size of the blob in MB and the max. time that we are prepared to wait to transfer 1MB. You can tweak timeoutInSecForOneMB variable to adjust the code to the network conditions. Any value that works for you when you deploy solution to Compute Emulator and use the Azure blob store will also work when you deploy to Azure Compute because you would use blob storage in the same datacenter and the latencies would be much lower.
foreach (CloudBlob blob in container.ListBlobs(new BlobRequestOptions() { UseFlatBlobListing = true }))
{
string blobName = blob.Uri.LocalPath.Substring(blob.Parent.Uri.LocalPath.Length);
if (blobName.Equals("command.bat"))
{
commandFileExists = true;
}
string file = Path.Combine(deployDir, blobName);
Trace.TraceInformation(System.DateTime.UtcNow.ToUniversalTime() + " Downloading blob " + blob.Uri.LocalPath + " from deployment container " + container.Name + " to file " + file);
//the time in seconds in which 1 MB of data should be downloaded (otherwise we timeout)
int timeoutInSecForOneMB = 5;
double sizeInMB = System.Math.Ceiling((double)blob.Properties.Length / (1024 * 1024));
// we set large enough timeout to download the blob of the given size
blob.DownloadToFile(file, new BlobRequestOptions() { Timeout = TimeSpan.FromSeconds(timeoutInSecForOneMB * sizeInMB) });
FileInfo fi = new FileInfo(file);
Trace.TraceInformation(System.DateTime.UtcNow.ToUniversalTime() + " Saved blob to file " + file + "(" + fi.Length + " bytes)");
}
After download has been completed the RunProcess() function spawns a child process to run the batch. The runtime process host WaIISHost.exe is configured to run elevated. This is done in the service definition file using the Runtime element. This is required so that the batch commands have admin rights (the process will run under Local System account)
<Runtime executionContext="elevated">
</Runtime>
Then we signal the wait handle (if implemented) and /or set busy flag that is used with the StatusCheck event to indicate configuration has completed and role can move to Ready state.
Trace.WriteLine("---------------Starting Deployment Batch Job--------------------");
if (commandFileExists)
RunProcess("cmd.exe", "/C command.bat");
else
Trace.TraceWarning("Command file command.bat was not found in container " + container.Name);
Trace.WriteLine(" Deployment completed.");
// signal role is Ready and should be included in the NLB rotation
this.busy = false;
//signal the wait handle to unblock OnStart() thread - if waiting has been implemented.
statusCheckWaitHandle.Set();
The sample solution includes a diagnostics.wadcfg file that transfers the traces and events to blob store (you want to delete or rename it once you go to production to avoid associated storage costs) every 60 seconds.
<?xml version="1.0" encoding="utf-8"?>
<DiagnosticMonitorConfiguration xmlns="https://schemas.microsoft.com/ServiceHosting/2010/10/DiagnosticsConfiguration" configurationChangePollInterval="PT5M" overallQuotaInMB="3005">
<Logs bufferQuotaInMB="250" scheduledTransferLogLevelFilter="Verbose" scheduledTransferPeriod="PT01M" />
<WindowsEventLog bufferQuotaInMB="50"
scheduledTransferLogLevelFilter="Verbose"
scheduledTransferPeriod="PT01M">
<DataSource name="System!*" />
<DataSource name="Application!*" />
</WindowsEventLog>
</DiagnosticMonitorConfiguration>
You can then deploy the solution to Azure Compute and look up the traces in the WADLogsTable using any Azure storage access tool such as Azure MMC.
Entering OnStart() - 03/06/2011 9:09:51
Blocking in OnStart() - waiting to be signalled.03/06/2011 9:09:51
OnStartInternal() - Started 03/06/2011 9:09:51
Waiting to complete configuration for this role .
---------------Downloading from Storage--------------------
03/06/2011 9:09:53 Downloading blob /contoso/command.bat from deployment container contoso to file C:\Users\micham\AppData\Local\dftmp\s0\deployment(688)\res\deployment(688).StorageDeployment.StorageDeployWebRole.0\directory\TempLocalStore\command.bat; TraceSource 'WaIISHost.exe' event
03/06/2011 9:09:53 Saved blob to file C:\Users\micham\AppData\Local\dftmp\s0\deployment(688)\res\deployment(688).StorageDeployment.StorageDeployWebRole.0\directory\TempLocalStore\command.bat(36 bytes); TraceSource 'WaIISHost.exe' event
03/06/2011 9:09:53 Downloading blob /contoso/vcredist_x64.exe from deployment container contoso to file C:\Users\micham\AppData\Local\dftmp\s0\deployment(688)\res\deployment(688).StorageDeployment.StorageDeployWebRole.0\directory\TempLocalStore\vcredist_x64.exe; TraceSource 'WaIISHost.exe' event
Waiting to complete configuration for this role .
Waiting to complete configuration for this role .
Waiting to complete configuration for this role .
Waiting to complete configuration for this role .
Waiting to complete configuration for this role .
03/06/2011 9:10:01 Saved blob to file C:\Users\micham\AppData\Local\dftmp\s0\deployment(688)\res\deployment(688).StorageDeployment.StorageDeployWebRole.0\directory\TempLocalStore\vcredist_x64.exe(5718872 bytes); TraceSource 'WaIISHost.exe' event
---------------Starting Deployment Batch Job--------------------
Information: Executing: cmd.exe
; TraceSource 'WaIISHost.exe' event
C:\Users\micham\AppData\Local\dftmp\s0\deployment(688)\res\deployment(688).StorageDeployment.StorageDeployWebRole.0\directory\TempLocalStore>vcredist_x64.exe /q ; TraceSource 'WaIISHost.exe' event
Waiting to complete configuration for this role .
Waiting to complete configuration for this role .
Waiting to complete configuration for this role .
Waiting to complete configuration for this role .
Waiting to complete configuration for this role .
Waiting to complete configuration for this role .
Waiting to complete configuration for this role .
Waiting to complete configuration for this role .
Waiting to complete configuration for this role .
; TraceSource 'WaIISHost.exe' event
; TraceSource 'WaIISHost.exe' event
C:\Users\micham\AppData\Local\dftmp\s0\deployment(688)\res\deployment(688).StorageDeployment.StorageDeployWebRole.0\directory\TempLocalStore>exit /b 0 ; TraceSource 'WaIISHost.exe' event
; TraceSource 'WaIISHost.exe' event
Information: Process Exit Code: 0
Process execution time: 13,151 seconds.
Deployment completed.
Exiting OnStart() - 03/06/2011 9:10:14
Information: Worker entry point called
Information: Working
If you need to redeploy the component you just upload the changed component versions to blob storage and restart your role to pick it up. When you combine this technique with the Web Deploy to update the code of Azure web role you should notice that the whole update process is faster and more flexible. This approach would be most useful in development and testing where otherwise you redeploy the solution multiple times.
You can download the solution here.