Azure Real World: Managing and Monitoring Drupal Sites on Windows Azure

A few weeks ago, I co-authored an article (with my colleague Rama Ramani) about how the Screen Actors Guild Awards website migrated its Drupal deployment from LAMP to Windows Azure: Azure Real World: Migrating a Drupal Site from LAMP to Windows Azure. Since then, Rama and another colleague, Jason Roth, have been working on writing up how the SAG Awards website was managed and monitored in Windows Azure. The article below is the fruit of their work…a very interesting/educational read.

Overview

Drupal is an open source content management system that runs on PHP. Windows Azure offers a flexible platform for hosting, managing, and scaling Drupal deployments. This paper focuses on an approach to host Drupal sites on Windows Azure, based on learning from a BPD Customer Programs Design Win engagement with the Screen Actors Guild Awards Drupal website. This paper covers guidelines and best practices for managing an existing Drupal web site in Windows Azure. For more information on how to migrate Drupal applications to Windows Azure, see Azure Real World: Migrating a Drupal Site from LAMP to Windows Azure.

The target audience for this paper is Drupal administrators who have some exposure to Windows Azure. More detailed pointers to Windows Azure content is provided throughout the paper as links.

Drupal Application Architecture on Windows Azure

Before reviewing the management and monitoring guidelines, it is important to understand the architecture of a typical Drupal deployment on Windows Azure. First, the following diagram displays the basic architecture of Drupal running on Windows and IIS7.

In the Windows Server scenario, you could have one or more machines hosting the web site in a farm. Those machines would either persist the site content to the file system or point to other network shares.

For Windows Azure, the basic architecture is the same, but there are some differences. In Windows Azure the site is hosted on a web role. A web role instance is hosted on a Windows Server 2008 virtual machine within the Windows Azure datacenter. Like the web farm, you can have multiple instances running the site. But there is no persistence guarantee for the data on the file system. Because of this, much of the shared site content should be stored in Windows Azure Blob storage. This allows them to be highly available and durable. Usually, a large portion of the site caters to static content which lends well to caching. And caching can be applied in a set of places – browser level caching, CDN to cache content in the edge closer to the browser clients, caching in Azure to reduce the load on backend, etc. Finally, the database can be located in SQL Azure. The following diagram shows these differences.

For monitoring and management, we will look at Drupal on Windows Azure from three perspectives:

• Availability: Ensure the web site does not go down and that all tiers are setup correctly. Apply best practices to ensure that the site is deployed across data centers and perform backup operations regularly.

• Scalability: Correctly handle changes in user load. Understand the performance characteristics of the site.

• Manageability: Correctly handle updates. Make code and site changes with no downtime when possible.

Although some management tasks span one or more of these categories, it is still helpful to discuss Drupal management on Windows Azure within these focus areas.

Availability

One main goal is that the Drupal site remains running and accessible to all end-users. This involves monitoring both the site and the SQL Azure database that the site depends on. In this section, we will briefly look at monitoring and backup tasks. Other crossover areas that affect availability will be discussed in the next section on scalability.

Monitoring

With any application, monitoring plays an important role with managing availability. Monitoring data can reveal whether users are successfully using the site or whether computing resources are meeting the demand. Other data reveals error counts and possibly points to issues in a specific tier of the deployment.

There are several monitoring tools that can be used.

• The Windows Azure Management Portal.

• Windows Azure diagnostic data.

• Custom monitoring scripts.

• System Center Operations Manager.

• Third party tools such as Azure Diagnostics Manager and Azure Storage Explorer.

The Windows Azure Management Portal can be used to ensure that your deployments are successful and running. You can also use the portal to manage features such as Remote Desktop so that you can directly connect to machines that are running the Drupal site.

Windows Azure diagnostics allows you to collect performance counters and logs off of the web role instances that are running the Drupal site. Although there are many options for configuring diagnostics in Azure, the best solution with Drupal is to use a diagnostics configuration file. The following configuration file demonstrates some basic performance counters that can monitor resources such as memory, processor utilization, and network bandwidth.

 <DiagnosticMonitorConfiguration xmlns="https://schemas.microsoft.com/ServiceHosting/2010/10/DiagnosticsConfiguration" 
                                 configurationChangePollInterval="PT1M" 
                                 overallQuotaInMB="4096">
  
   <DiagnosticInfrastructureLogs bufferQuotaInMB="10"
                                 scheduledTransferLogLevelFilter="Error"
                                 scheduledTransferPeriod="PT1M" />
  
   <Logs bufferQuotaInMB="0"
         scheduledTransferLogLevelFilter="Verbose"
         scheduledTransferPeriod="PT1M" />
  
   <Directories bufferQuotaInMB="0" scheduledTransferPeriod="PT5M">
     <!-- These three elements specify the special directories that are set up for the log types -->
     <CrashDumps container="wad-crash-dumps" directoryQuotaInMB="256" />
     <FailedRequestLogs container="wad-frq" directoryQuotaInMB="256" />
     <IISLogs container="wad-iis" directoryQuotaInMB="256" />
   </Directories>
  
   <PerformanceCounters bufferQuotaInMB="0" scheduledTransferPeriod="PT1M">
     <!-- The counter specifier is in the same format as the imperative diagnostics configuration API -->
     <PerformanceCounterConfiguration counterSpecifier="\Memory\Available MBytes" sampleRate="PT10M" />
     <PerformanceCounterConfiguration counterSpecifier="\Network Interface(*)\Bytes Total/sec" sampleRate="PT10M" />
     <PerformanceCounterConfiguration counterSpecifier="\Processor(*)\% Processor Time" sampleRate="PT10M" />
     <PerformanceCounterConfiguration counterSpecifier="\System\Processor Queue Length" sampleRate="PT10M" />
     <PerformanceCounterConfiguration counterSpecifier="\Web Service(_Total)\Current Connections" sampleRate="PT10M" />
     <PerformanceCounterConfiguration counterSpecifier="\Web Service(_Total)\Bytes Total/sec" sampleRate="PT10M" />
     <PerformanceCounterConfiguration counterSpecifier="\Web Service(_Total)\Connection Attempts/sec" sampleRate="PT10M" />
     <PerformanceCounterConfiguration counterSpecifier="\Web Service(_Total)\Get Requests/sec" sampleRate="PT10M" />
     <PerformanceCounterConfiguration counterSpecifier="\Web Service(_Total)\Files Sent/sec" sampleRate="PT10M" />
   </PerformanceCounters>
  
   <WindowsEventLog bufferQuotaInMB="0"
                    scheduledTransferLogLevelFilter="Verbose"
                    scheduledTransferPeriod="PT5M">
  
     <!-- The event log name is in the same format as the imperative diagnostics configuration API -->
  
     <DataSource name="System!*" />
  
   </WindowsEventLog>
  
 </DiagnosticMonitorConfiguration>

For more information about setting up diagnostic configuration files, see How to Use the Windows Azure Diagnostics Configuration File. This information is stored locally on each role instance and then transferred to Windows Azure storage per a defined schedule or on-demand. See Getting Started with Storing and Viewing Diagnostic Data in Windows Azure Storage. Various monitoring tools, such as Azure Diagnostics Manager, help you to more easily analyze diagnostic data.

Monitoring the performance of the machines hosting the Drupal site is only part of the story. In order to plan properly for both availability and scalability, you should also monitor site traffic, including user load patterns and trends. Standard and custom diagnostic data could contribute to this, but there are also third-party tools that monitor web traffic. For example, if you know that spikes occur in your application during certain days of the week, you could make changes to the application to handle the additional load and increase the availability of the Drupal solution.

Backup Tasks

To remain highly available, it is important to backup your data as a defense-in-depth strategy for disaster recovery. This is true even though SQL Azure and Windows Azure Storage both implement redundancy to prevent data loss. One obvious reason is that these services cannot prevent administrator error if data is accidentally deleted or incorrectly changed.

SQL Azure does not currently have a formal backup technology, although there are many third-party tools and solutions that provide this capability. Usually the database size for a Drupal site is relatively small. In the case of SAG Awards, it was only ~100-150 MB. So performing an entire backup using any strategy was relatively fast. If your database is much larger, you might have to test various backup strategies to find the one that works best.

Apart from third-party SQL Azure backup solutions, there are several strategies for obtaining a backup of your data:

· Use the Drush tool and the portabledb-export command.

· Periodically copy the database using the CREATE DATABASE Transact-SQL command.

· Use Data-tier applications (DAC) to assist with backup and restore of the database.

SQL Azure backup and data security techniques are described in more detail in the topic, Business Continuity in SQL Azure.

Note that bandwidth costs accrue with any backup operation that transfers information outside of the Windows Azure datacenter. To reduce costs, you can copy the database to a database within the same datacenter. Or you can export the data-tier applications to blob storage in the same datacenter.

Another potential backup task involves the files in Blob storage. If you keep a master copy of all media files uploaded to Blob storage, then you already have an on-premises backup of those files. However, if multiple administrators are loading files into Blob storage for use on the Drupal site, it is a good idea to enumerate the storage account and to download any new files to a central location. The following PHP script demonstrates how this can be done by backing up all files in Blob storage after a specified modification date.

 <?php
 /**
 * Backup the blob storage to a local folder 
 * 
 * PHP version 5.3
 * 
 * @category Blob Backup
 * @package Windows Azure
 * @author Bibin Kurian
 * @copyright 2011 Microsoft Corp. (https://www.microsoft.com)
 * @license New BSD license, (https://www.opensource.org/licenses/bsd-license.php)
 * @version SVN: 1.0
 * @link
 * 
 */
  
 /** Microsoft_WindowsAzure_Storage_Blob */
 require_once 'Microsoft/WindowsAzure/Storage/Blob.php';
  
 //Windows Azure BLOB storage account name
 define("AZURE_STORAGE_ACCOUNT", "YOUR_ACCOUNTNAME");
  
 //Windows Azure BLOB storage container name
 define("AZURE_STORAGE_CONTAINER", "YOUR_STORAGECONTAINERNAME");
  
 //Windows Azure BLOB storage secret key
 define(
     "AZURE_STORAGE_KEY", 
     "YOUR_SECRETKEY"
 );
  
 //backup folder
 define("STORAGE_BACKUP_DIR", "YOUR_LOCALDRIVE");
  
 //backup from date
 define("DEFAULT_BACKUP_FROM_DATE", strtotime("Mon, 19 Dec 2011 22:00:00 GMT"));
  
 //backup to date
 define("DEFAULT_BACKUP_TO_DATE", time());
  
 //directory seperator
 define("DS", "\\");
  
 //start backup logic
  
 //current datetime to create the backup folder
 $now = date("F j, Y, g.i A");
 $fullBackupDirPath = STORAGE_BACKUP_DIR . DS . $now . DS. AZURE_STORAGE_CONTAINER;
 mkdir($fullBackupDirPath, 0755, true);
  
 //For the directory creations, pointing the current directory to user specified directory
 chdir($fullBackupDirPath);
  
 //BLOB object
 $blobObj = new Microsoft_WindowsAzure_Storage_Blob(
     'blob.core.windows.net', 
     AZURE_STORAGE_ACCOUNT,
     AZURE_STORAGE_KEY
 );
  
 //$blobObj->setProxy(true, 'YOUR_PROXY_IF_NEEDED', 80);
  
 $blobs = (array)$blobObj->listBlobs(AZURE_STORAGE_CONTAINER, '', '', 35000);
 backupBlobs($blobs, $blobObj);
 function backupBlobs($blobs, $blobObj) {
     foreach ($blobs as $blob) {
         if (strtotime($blob->lastmodified) >= DEFAULT_BACKUP_FROM_DATE && strtotime($blob->lastmodified) <= DEFAULT_BACKUP_TO_DATE) {
             $path = pathinfo($blob->name);
             if ($path['basename'] != '$$$.$$$') {
                 $dir = $path['dirname'];
                 $oldDir = getcwd();
                 if (handleDirectory($dir)) {
                     chdir($dir);
                     $blobObj->getBlob(
                         AZURE_STORAGE_CONTAINER,
                         $blob->name,
                         $path['basename']
                     );
                     chdir($oldDir);
                 }
             }
         }
     }
 }
  
 function handleDirectory($dir) {
     if (!checkDirExists($dir)) {
         return mkdir($dir, 0755, true);
     }
     return true;
 }
  
 function checkDirExists($dir) {
     if(file_exists($dir) && is_dir($dir)) {
         return true;
     }
     return false;
 }
 ?>

This script has a dependency on the Windows Azure SDK for PHP. Also note there are several parameters that you must modify such as the storage account, secret, and backup location. As with SQL Azure, bandwidth and transaction charges apply to a backup script like this.

Scalability

Drupal sites on Windows Azure can scale as load increased through typical strategies of scale-up, scale-out, and caching. The following sections describe the specifics of how these strategies are implemented in Windows Azure.

Typically you make scalability decisions based on monitoring and capacity planning. Monitoring can be done in staging during testing or in production with real-time load. Capacity planning factors in projections for changes in user demand.

Scale Up

When you configure your web role prior to deployment, you have the option of specifying the Virtual Machine (VM) size, such as Small or ExtraLarge. Each size tier adds additional memory, processing power, and network bandwidth to each instance of your web role. For cost efficiency and smaller units of scale, you can test your application under expected load to find the smallest virtual machine size that meets your requirements.

The workload usually in most popular Drupal websites can be separated out into a limited set of Drupal admins making content changes and a large user base who perform mostly read-only workload. End users can be allowed to make ‘writes’, such as uploading blogs or posting in forums, but those changes are not ‘content changes’. Drupal admins are setup to operate without caching so that the writes are made directly to SQL Azure or the corresponding backend database. This workload performs well with Large or ExtraLarge VM sizes. Also, note that the VM size is closely tied to all hardware resources, so if there are many content-rich pages that are streaming content, then the VM size requirements are higher.

To make changes to the Virtual Machine size setting, you must change the vmsize attribute of the WebRole element in the service definition file, ServiceDefinition.csdef. A virtual machine size change requires existing applications to be redeployed.

Scale Out

In addition to the size of each web role instance, you can increase or decrease the number of instances that are running the Drupal site. This spreads the web requests across more servers, enabling the site to handle more users. To change the number of running instances of your web role, see How to Scale Applications by Increasing or Decreasing the Number of Role Instances.

Note that some configuration changes can cause your existing web role instances to recycle. You can choose to handle this situation by applying the configuration change and continue running. This is done by handling the RoleEnvironment.Changing event. For more information see, How to Use the RoleEnvironment.Changing Event.

A common question for any Windows Azure solution is whether there is some type of built-in automatic scaling. Windows Azure does not provide a service that provides auto-scaling. However, it is possible to create a custom solution that scales Azure services using the Service Management API. For an example of this approach, see An Auto-Scaling Module for PHP Applications in Windows Azure.

Caching

Caching is an important strategy for scaling Drupal applications on Windows Azure. One reason for this is that SQL Azure implements throttling mechanisms to regulate the load on any one database in the cloud. Code that uses SQL Azure should have robust error handling and retry logic to account for this. For more information, see Error Messages (SQL Azure Database). Because of the potential for load-related throttling as well as for general performance improvement, it is strongly recommended to use caching.

Although Windows Azure provides a Caching service, this service does not currently have interoperability with PHP. Because of this, the best solution for caching in Drupal is to use a module that uses an open-source caching technology, such as Memcached.

Outside of a specific Drupal module, you can also configure Memcached to work in PHP for Windows Azure. For more information, see Running Memcached on Windows Azure for PHP. Here is also an example of how to get Memcached working in Windows Azure using a plugin: Windows Azure Memcached plugin.

In a future paper, we hope to cover this architecture in more detail. For now, here are several design and management considerations related to caching.

Area	Consideration
Design and Implementation	For a technology like Memcached, will the cache be collocated (spread across all web role instances)? Or will you attempt to setup a dedicated cache ring with worker roles that only run Memcached?
Configuration	What memory is required and how will items in the cache be invalidated?
Performance and Monitoring	What mechanisms will be used to detect the performance and overall health of the cache?

For ease of use and cost savings, collocation of the cache across the web role instances of the Drupal site works best. However, this assumes that there is available reserve memory on each instance to apply toward caching. It is possible to increase the virtual machine size setting to increase the amount of available memory on each machine. It is also possible to add additional web role instances to add to the overall memory of the cache while at the same time improving the ability of the web site to respond to load. It is possible to create a dedicated cache cluster in the cloud, but the steps for this are beyond the scope of this paper[RR1] .

For Windows Azure Blob storage, there is also a caching feature built into the service called the Content Delivery Network (CDN). CDN provides high-bandwidth access to files in Blob storage by caching copies of the files in edge nodes around the world. Even within a single geographic region, you could see performance improvements as there are many more edge nodes than Windows Azure datacenters. For more information, see Delivering High-Bandwidth Content with the Windows Azure CDN.

Manageability

It is important to note that each hosted service has a Staging environment and a Production environment. This can be used to manage deployments, because you can load and test and application in staging before performing a VIP swap with production.

From a manageability standpoint, Drupal has an advantage on Windows Azure in the way that site content is stored. Because the data necessary to serve pages is stored in the database and blob storage, there is no need to redeploy the application to change the content of the site.

Another best practice is to use a separate storage account for diagnostic data than the one that is used for the application itself. This can improve performance and also helps to separate the cost of diagnostic monitoring from the cost of the running application.

As mentioned previously, there are several tools that can assist with managing Windows Azure applications. The following table summarizes a few of these choices.

Tool	Description
Windows Azure Management Portal	The web interface of the Windows Azure management portal shows deployments, instance counts and properties, and supports many different common management and monitoring tasks.
Azure Diagnostics Manager	A Red Gate Software product that provides advanced monitoring and management of diagnostic data. This tool can be very useful for easily analyzing the performance of the Drupal site to determine appropriate scaling decisions.
Azure Storage Explorer	A tool created by Neudesic for viewing Windows Azure storage account. This can be useful for viewing both diagnostic data and the files in Blob storage.

Additional Drupal Resources

The following list contains additional resources for managing your Drupal deployment on Windows Azure:

• Install Drupal for Windows
• Azure Real World: Migrating a Drupal Site from LAMP to Windows Azure
• Drupal on Windows Azure
• Getting Started with Drupal 7 Administration
• Drupal Statistics
• Windows Azure Traffic Manager (WATM)
• System Center Operations Manager MP for Azure applications
• System Center Operations Manager MP for SQL Azure

Share this on Twitter