Supporting Billions of entities/rows for Mobile – Android Series - Part 4–Building a Cloud-based RESTful service for our Android, iOS, and Windows Phone 7 Clients

Previous Posts to this post

Previous Blog Posts Link Supporting Billions of entities/rows for Mobile – Android Series - Part 1 - Why Scale Matters https://blogs.msdn.com/b/brunoterkaly/archive/2011/09/27/supporting-billions-of-entities-rows-for-mobile-android-series-part-1-why-scale-matters.aspx Supporting Billions of entities/rows for Mobile – Android Series - Part 2 - What are some high level cloud offerings? https://blogs.msdn.com/b/brunoterkaly/archive/2011/09/27/supporting-billions-of-entities-rows-for-mobile-android-series-part-2-what-are-some-high-level-cloud-offerings.aspx Supporting Billions of entities/rows for Mobile – Android Series - Part 3–Architecture and Data Options https://blogs.msdn.com/b/brunoterkaly/archive/2011/09/28/supporting-billions-of-entities-rows-for-mobile-android-series-part-3-architecture-and-data-options.aspx You will need to download the free Azure trial.

What this post will cover

There are two parts: Part 1 Setting up your hardware with software Part 2 Writing the RESTful Service that will support Android clients

Part 1 - Needed Software and Hardware

Software To have a valid copy of Visual Studio 2010 Professional, Premium, Ultimate ,or Test Professional, including SQL Server 2008 Express. Preferably Windows 7 Professional, Enterprise, or Ultimate. I haven’t tested Vista or Visual Studio 2008. The Azure SDK - link at top of blog. I am running Windows 7 Ultimate – 64-bit Hardware 1.6 GHz or faster non-Mac CPU 2+ GB RAM 20 GB Hard Disk

Part 1 - Setup instructions for the Azure SDK

Details for this instructions can be found here:   See section: Standalone Windows Azure SDK Instructions for installing Azure SDK can be found here

Part 1 – Verifying your setup

Video that will help you verify your setup

Part 2 – Why REST?

Usually the debate is whether to use REST or SOAP. Here are some images to help you understand the differences. Some more differences: Here are some other points to remember: REST is an architecture style An architectural style is a set of constraints that can be applied when building something. Architecture can guide you Architecture describes the features that can be used to guide the implementation of a software system. Clients request resources from endpoints REST is an architectural style that can be used to build software in which clients (user agents) can make requests of services (endpoints). REST is client-server REST is one way to implement a client-server architectural style—in fact, REST explicitly builds on the client-server architectural style. Roy Fielding REST is based on a dissertation found here   Roy's Dissertation

Part 2 – New Cloud Project

Step Task 1 Create a new cloud project with Visual Studio as Administrator.     Running as Administrator, Select File, New Project. The project type is Windows Azure Project. At this point, you should be selecting the Cloud template and Windows Azure Project Visual C# . For any of this to work, you need to have installed the Azure SDK as stated above. Notice I have also provided a Name, Location, and Solution Name. We are about to add a Web Role to our project. Think of the Web Role as the container for our REST-based service. We will add an .svc file to our Web Role. The .svc file will be what our Android application communicates with to retrieve data from our service.  Visual Studio makes adding a Web Role easy. There are several types of Web Roles. For this demo we will select ASP.NET Web Role.   Go ahead and select ASP.NET Web Role so that your screen looks like this: Rename your WebRole to something more appropriate. I renamed it to FastMotorcycleList_WebRole. Here is what Solution Explorer should look like: When you create a new Windows Azure Project, it is actually composed of two separate projects. There are a couple of things to notice above about the Configuration Project and the RESTful Service Project: Configuration Project This is the where you specify the runtime settings for your project when it runs. For example, you specify the number of instances you would like. Compute instance is a VM dedicated to your service. This is how you control scale - more users means you should allocate more compute instances. As with other cloud-computing platforms the primary scalability mechanism is horizontal scaling through scaling out the number of compute instances (or VMs) on which the service runs. When you increase the instance count to 2, your service will be running on 2 VMs. REST Service Project This is the project that will contain our RESTful service. Notice that there are a lot of files that have been given to us by the project creation wizard in Visual Studio. We will add some files as well as removing files. Most of the work in this post will take place here.   Quick Note   It is possible to programmatically change the instance count using external applications. Be aware that there is a REST based Service Management API that lets you change the instance count programmatically. You can download an example of this from CodePlex.

Part 2 – Eliminating Files We Don’t Need and Testing our Configuration of Visual Studio

It turns out this will be an extremely simple Web Role. We will eliminate many of the files that aren’t needed. To keep this demo as simple as possible, we are not authenticating or authorizing users. Remove the Account folder and the About.aspx file. We’ll keep Default.aspx around simply to be able to run our blank project to make sure everything is setup correctly. Before After At this point we are ready to run our project. It is by no means finished. We should just run to make sure our everything is setup correctly. We cannot move forward until we validate that the storage and compute emulators work correctly. Right mouse click on the solution as follows: Assuming you do not get compiler errors (which you shouldn’t), you are ready to run. Go to the Debug menu and select Start Debugging. You should be able to see the storage and compute emulators. Before you see default.aspx pop up, the emulation environment needs to start. The emulation environment is very useful, because it allows you to run your Azure project locally, without having to deploy the application to the Azure Data centers. Typically, you can do all your testing and debugging without the hassles of deployment. There are exceptions to testing everything locally. Windows Azure compute emulator A tool that locally emulates the environment in which hosted services run. Windows Azure storage emulator A tool that locally emulates the services of Windows Azure storage.   Understanding Emulators   You can start the emulation environment manually – if you wish. This link explains how to start the emulation environments outside of starting Visual Studio.   Azure SDK Tools Your running app should like this:

Step 2 – Adding startup code

We will need to write some startup code that kicks off once our RESTful service starts. This startup code needs to prepare our Azure Table store. We talked about Azure Tables in a previous post. This startup code typically goes in global.asax.cs.

global.asax.cs

This is your startup code.

using System;using System.Collections.Generic;using System.Linq;using System.Web;using System.Web.Security;using System.Web.SessionState;// These 3 using statements are needed for the Application_Start() methodusing Microsoft.WindowsAzure.ServiceRuntime;using Microsoft.WindowsAzure;using Microsoft.WindowsAzure.StorageClient;namespace FastMotorcycleList_WebRole{ public class Global : System.Web.HttpApplication { protected void Application_Start(object sender, EventArgs e) { // Grab the configuration string string configValue = RoleEnvironment.GetConfigurationSettingValue("DataConnectionString"); // The account key will map to: // (1) development storage // (2) our storage account in a data center var account = CloudStorageAccount.Parse(configValue); // Create CloudTableClient tableClient = account.CreateCloudTableClient(); tableClient.CreateTableIfNotExist("lists"); } }}

Comment

The code above is needed to setup the table objects that will be used to store data. CloudTableClient is part of the Azure SDK. It provides a client for accessing the Windows Azure Table service. It allows you to create tables, delete tables, list tables, etc.

Step 2 – Adding a DataConnectionString

The point here is to setup a DataConnectionString, which will be used to either direct our application to run in a Microsoft Data Center or run in our local development fabric. Cloud-based in a Microsoft Data Center This is a future step. We will deploy our application in a Microsoft Data Center when we are finished writing, debugging and testing our application. Storage Emulator/Compute Emulator When developing our application, it is possible to run in a local simulation environment. This is very convenient. It saves us the step of deploying our application to test it. We can run our web role and levage our table objects all from the comfort of our local development computer. In the system tray you can see the Compute Emulator. Start by selecting properties as seen below. Click in a step-by-step fashion as seen below. The end result is that we want a setting called DataConnectionString that has a value of UseDevelpmentStorage=true.

Step 2 – Adding diagnostic code

We need to add code to WebRole.cs. DiagnosticMonitorConfiguration is part of the Azure SDK that allows you to track diagnostic information as the application runs. It represents set of data buffers for logging. You can find more here.   DiagnosticMonitorConfiguration

WebRole.cs

Add the following code to WebRole.cs

using System;using System.Collections.Generic;using System.Linq;using Microsoft.WindowsAzure;using Microsoft.WindowsAzure.Diagnostics;using Microsoft.WindowsAzure.ServiceRuntime;namespace FastMotorcycleList_WebRole{ public class WebRole : RoleEntryPoint { public override bool OnStart() { DiagnosticMonitorConfiguration diagnosticConfig = DiagnosticMonitor.GetDefaultInitialConfiguration(); diagnosticConfig.Directories.ScheduledTransferPeriod = TimeSpan.FromMinutes(1); diagnosticConfig.Directories.DataSources.Add( AzureLocalStorageTraceListener.GetLogDirectory()); return base.OnStart(); } }}

Comments

The main task left is to add AzureLocalStorageTraceListener, which is a custom class that defines aspects of the trace file, such as: 1. The name of the trace file 2. The size of the trace file 3. The name of the container of the trace file 4. The path of the trace file

AzureLocalStorageTraceListener

This is a custom class that we are adding. It provides details about the tracing information that we are tracking. Call the file AzureLocalStorageTraceListener.cs.

using System;using System.Collections.Generic;using System.Linq;using System.Web;// These using statements need to be added// based on the code belowusing System.Diagnostics;using System.IO;using Microsoft.WindowsAzure.Diagnostics;using Microsoft.WindowsAzure.ServiceRuntime;namespace FastMotorcycleList_WebRole{ public class AzureLocalStorageTraceListener : XmlWriterTraceListener { public AzureLocalStorageTraceListener() : base(Path.Combine(AzureLocalStorageTraceListener.GetLogDirectory().Path, "FastMotorcycleList.Web.svclog")) { } public static DirectoryConfiguration GetLogDirectory() { DirectoryConfiguration directory = new DirectoryConfiguration(); directory.Container = "mobile-tracefiles"; directory.DirectoryQuotaInMB = 10; directory.Path = RoleEnvironment.GetLocalResource("FastMotorcycleList.Web.svclog").RootPath; return directory; } }}

Step 2 – Specifying local storage for FastMotorcycleList.Web.svclog

You will need to setup Project Properties to finish this task. You will need to click on Local Storage, followed by Add Local Storage. Next you will enter: Filename FastMotorcyleList.Web.svclog Size 10 Here is what the finished work looks like:

Comments

1. Used by code in WebRole.cs 2. Sets up our trace file to help us with diagnostics 3. The name of the trace file is FastMotorcycleList.Web.svclog 4. Inherits from XMLWriteTraceListener to format the trace entires in XML

Step 2 - Review

This is just a quick recap. Now we are ready to start the main RESTful service aspect of our project. This next section is very important.

Step 2 – Adding our RESTful service

We will now add the code that allows Android clients (or any client for that matter) to connect and perform CRUD operations (create, read, update, delete) against our table objects. To start things off, let’s add a data provider, which is a class that abstracts away the details CRUD operations and details of the data store. The following 3 classes will be contained inside of FastMotorcycleListDataProvider.cs file. In Solution Explorer, add a new class by right mouse clicking on FastMotorcycleList_WebRole and call it FastMotorcycleListDataProvider.cs. Call the file FastMotorcycleListDataProvider.cs. Your Solution Explorer should look like this:

FastMotorcycleListDataProvider.cs

You will now paste in the code for this file (FastMotorcycleListDataProvider.cs). There are 3 main classes here, although the class FastMotorcycleListDataProvider is the only one that is used by the RESTful service.

using System;using System.Collections.Generic;using System.Linq;using System.Web;// Added using statementsusing Microsoft.WindowsAzure;using Microsoft.WindowsAzure.ServiceRuntime;using Microsoft.WindowsAzure.StorageClient;using System.Data.Services.Client;namespace FastMotorcycleList_WebRole{ // --------------------------------------------------------- // Purpose of Class: Provides basic create, read, delete operations to the service. // Abstracts away details of interacting with Azure Tables. // Notes: Uses FastMotorcycleList as persistant list public class FastMotorcycleListDataProvider { private FastMotorcycleList _list; // --------------------------------------------------------- // Purpose of Method: Connects to Azure Table Service and retrieves data // Notes: Uses web.config to read account credentials. // Developer needs to edit web.config to either point // to Azure Tables in local development environment // or up in cloud. public FastMotorcycleListDataProvider() { string configValue = RoleEnvironment.GetConfigurationSettingValue("DataConnectionString"); var account = CloudStorageAccount.Parse(configValue); _list = new FastMotorcycleList(account.TableEndpoint.ToString(), account.Credentials); } // --------------------------------------------------------- // Purpose of Method: Retrieves list from Azure Table Service // Notes: Uses "owner" for lookup // Owner is part of partitionkey so retrieval is fast public List<string> GetItems(string owner) { var results = from entity in _list.Items where entity.PartitionKey == owner select entity; var list = new List<string>(); foreach (var item in results) { list.Add(item.RowKey); } return list; } // --------------------------------------------------------- // Purpose of Method: Adds an entity to Azure Table Service // Notes: Each entity is associated with an owner public void AddItem(string owner, string item) { _list.AddObject("lists", new FastMotorcycleListItem(owner, item)); _list.SaveChanges(); } // --------------------------------------------------------- // Purpose of Method: Deletes an entity to Azure Table Service // Notes: Each entity is associated with an owner // --------------------------------------------------------- public void DeleteItem(string owner, string item) { var entity = (from i in _list.Items where i.PartitionKey == owner && i.RowKey == item select i).Single(); _list.DeleteObject(entity); _list.SaveChanges(); } } // --------------------------------------------------------- // Purpose of Class: Acts as the blueprint for an entity // Notes: The properties you add here define the // name value pairs that make up the table object public class FastMotorcycleListItem : TableServiceEntity { // --------------------------------------------------------- // Purpose of Method: Default constructor public FastMotorcycleListItem() { } // --------------------------------------------------------- // Purpose of Method: Represents the entity of 2 columns // (owner and list item) // Notes: No other properties defined because // only a two column table public FastMotorcycleListItem(string partitionKey, string rowKey) : base(partitionKey, rowKey) { } } // --------------------------------------------------------- // Purpose of Class: Abstracts away the fact that our Azure table // is in the cloud. TableServiceContext allows // us to code the entity as though it were // local public class FastMotorcycleList : TableServiceContext { // --------------------------------------------------------- // Purpose of Method: Connect to Azure Table Service // using storage credentials public FastMotorcycleList(string baseAddress, StorageCredentials storageCredentials) : base(baseAddress, storageCredentials) { } // --------------------------------------------------------- // Purpose of Method: Our list of entities is called "lists" public DataServiceQuery<FastMotorcycleListItem> Items { get { return this.CreateQuery<FastMotorcycleListItem>("lists"); } } }}

Comments

1. There are 3 classes but only FastMotorcycleListDataProvider is used outside this class. 2. Together these 3 classes allow us to communicate with the Azure Table Service. 3. Row key and partition key is very important and determines how your data gets scaled across multiple storage nodes. Read the note below.

Partition Key and Row Key

This is a very important concept. This is an important read for those wanting more details: The important points relate to partitionkey and rowkey. Notice that there are 2 partitions. Each partition is defined by the partitionkey and rowkey. Partition 1 and Partition 2 are 2 separate storage nodes, on separate physical machines. The storage system achieves good scalability by distributing the partitions across many storage nodes. The system monitors the usage patterns of the partitions, and automatically balances these partitions across all the storage nodes.

Step 2 – Explanation of the 3 classes

Here are the 3 main classes inside of FastMotorcycleListDataProvider.

Class Subclasses Description class FastMotorcycleListDataProvider none This is the main class used by the rest service to perform REST operations. Contains: 1. GetItems() 2. AddItem() 3. DeleteItem() class FastMotorcycleListItem : TableServiceEntity TableServiceEntity Represents one item in the list of items tracked in the Azure tables. Defines the partition and row key and any other data elements we wish to persist inside of our table objects. TableServiceEntity represents a set of public properties among which must be the PartitionKey and RowKey, which are the primary key of the Azure Table. Read more here. class FastMotorcycleList : TableServiceContext TableServiceContext Represents an Azure table, composed of multiple FastMotorCycleListItems. At a lower level, represents a DataServiceContext object and allows the developer to use object oriented techniques to communicate with a table inside the Windows Azure Table service. DataServiceContext to serve as a data access context class. DataServiceContext supports functionality such as AddObject() to add an entity, UpdateObject() to update an entity, SaveChanges() to save changes to an entity, and CreateQuery<T> to create a query against the dataset. The core functionality is supported in both synchronous and asynchronous versions. DataServiceContext also supports batch operations.       Methods for FastMotorcycleListDataProvider public List<string> GetItems(string owner) Retrieves a list of fast motorcycles, given an owner. public void AddItem(string owner, string item) Adds a fast motorcycle to the list. public void DeleteItem(string owner, string item) Deletes a fast motorcycle to the list.

Step 2 – Adding references

Adding a reference to System.Data.Services.Client. Selecting the reference.

Step 2 – Adding RESTful Code

At this point we are ready to start the main piece of the RESTful service. The Android Application will make a all into FastMotorcycleListService.svc. From there calls will be made into the classes inside of FastMotorcycleListDataProvider. The actual interaction with the underlying table is abstracted away in FastMotorcycleListDataProvider, which makes use of two helper classes: FastMotorcycleList Represents our list of fast motorcycles in Azure tables FastMotorcycleListItem Represents an individual entity of a fast motorcycle in Azure tables The next step here is to finish the RESTful Service. There are two files to edit here. The .svc file is edited by writing XML while the second file ( .cs) is edited using the built-in C# code editor. FastMotorcycleListService.svc will be edited to use a Web Service Host Factory. Using this approach a host instance is created dynamically in response to incoming messages. The managed hosting environments that support dynamic activation are Internet Information Services (IIS) and Windows Process Activation Service (WAS), which is supported in Windows Azure.

FastMotorcycleListService.svc

The following markup will automatically instantiate an object of type FastMotorcycleListService. It is this code that will begin to service requests from mobile devices.

<%@ ServiceHost Debug="true" Factory="System.ServiceModel.Activation.WebServiceHostFactory"Service="FastMotorcycleList_WebRole.FastMotorcycleListService" %>

Step 2 – Writing the code-behind

Let’s turn our attention to FastMotorcycleListService.svc.cs.

FastMotorcycleListService.svc.cs

The following markup will automatically instantiate an object of type FastMotorcycleListService. It is this code that will begin to service requests from mobile devices. One important thing to notice here is that the data coming back is in JSON format. You could also use XML format. Read more here:

using System;using System.Collections.Generic;using System.Linq;using System.Web;using System.ServiceModel;using System.ServiceModel.Web;namespace FastMotorcycleList_WebRole{ [ServiceContract] // --------------------------------------------------------- // Purpose of Class: The main class to service mobile clients RESTfully // // Notes: Uses FastMotorcycleListDataProvider public class FastMotorcycleListService { // --------------------------------------------------------- // Purpose: Helper class to abstract away the underlying // details of the Azure Table Service. private FastMotorcycleListDataProvider _data; // --------------------------------------------------------- // Purpose: Constructor for abstracted Azure Table Service public FastMotorcycleListService() { _data = new FastMotorcycleListDataProvider(); } // --------------------------------------------------------- // Purpose: Retrieve a list of Fast Motorcycles based on list // owner [OperationContract] [WebGet(UriTemplate = "/list/{owner}", ResponseFormat = WebMessageFormat.Json)] public List<string> GetItems(string owner) { return _data.GetItems(owner); } // --------------------------------------------------------- // Purpose: Add a new Fast Motorcycle based on owner [OperationContract] [WebInvoke(UriTemplate = "/list/{owner}", Method = "POST", RequestFormat = WebMessageFormat.Json)] public void AddItem(string owner, string item) { _data.AddItem(owner, item); } // --------------------------------------------------------- // Purpose: Delete a Fast Motorcycle based on Owner [OperationContract] [WebInvoke(UriTemplate = "/list/{owner}/{item}", Method = "DELETE")] public void DeleteItem(string owner, string item) { _data.DeleteItem(owner, item); } }}

Step 2 – Redefining Web.Config

we need to change the Web.config that was generated when we first created our cloud-based project. We will simplify this file. We will erase the existing web.config file and replace it with new configuration code. This is essentially a configure-less configuration file. Instead, we are using Web Serivce Host Factory, as explained earlier. Start by replace web.config with this code:

<?xml version="1.0"?><configuration> <configSections></configSections> <!--To collect diagnostic traces, uncomment the section below. To persist the traces to storage, update the DiagnosticsConnectionString setting with your storage credentials. To avoid performance degradation, remember to disable tracing on production deployments. <system.diagnostics> <sharedListeners> <add name="AzureLocalStorage" type="AzureShoppingList.Web.AzureLocalStorageTraceListener, AzureShoppingList.Web"/> </sharedListeners> <sources> <source name="System.ServiceModel" switchValue="Verbose, ActivityTracing"> <listeners> <add name="AzureLocalStorage"/> </listeners> </source> <source name="System.ServiceModel.MessageLogging" switchValue="Verbose"> <listeners> <add name="AzureLocalStorage"/> </listeners> </source> </sources> </system.diagnostics> --> <system.diagnostics> <trace> <listeners> <add type="Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitorTraceListener, Microsoft.WindowsAzure.Diagnostics, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" name="AzureDiagnostics"> <filter type=""/> </add> </listeners> </trace> </system.diagnostics> <system.web> <compilation debug="true" targetFramework="4.0"/> </system.web> <system.webServer> <modules runAllManagedModulesForAllRequests="true"/> </system.webServer></configuration>

Step 2 – Completed

We should now be able to compile the project successfully. The output window should be error free.

Next Step

Before constructing our RESTful client (either Android, iOS, Windows Phone 7), we will test with a browser. The whole point of REST is that it is built upon http. The next blog post will explore using Fiddler to test our RESTful service.