Deploying an HPC Cluster using just PowerShell (Part I)
Last month my colleague Don Pattee at the HPC team announced the availability of the Windows Azure HPC Scheduler and HPC pack 2008 R2 Service pack 3 releases. The title is long, but the story is simple, while we updated our HPC on-premise solution with further enhancements, we released our first version of the Azure HPC Scheduler! That means you can deploy a cluster without having to provision a single computer of your own; when you are done with the cluster, simply delete it and the Billing stops. This is profoundly Convenient. For this release, we also added a web portal which users can share/submit/monitor jobs. In addition, we provide REST based interface so that applications can be designed to utilized the cluster over the internet securely - Confidence. Convenience and Confidence are often the key ingredients to foster Collaboration; the ease of sharing simulations & data on public cloud will make teaching those old HPC apps a few new tricks easier. In other words, creating HPC Applications as a service will be much easier.
To get started, you definitely don’t want to miss Alice’s videos on a quick overview & publishing a sample service. Windows Azure Platform Training kit has been updated at: https://www.microsoft.com/download/en/details.aspx?id=8396 under the HPC section. There are 3 new samples + 2 new demos for you to jump start a new HPC project.
If you have watched Alice’s videos, you’ll notice the Windows Azure HPC Scheduler must be deployed with Visual Studio. However, that is no longer the only way of making a deployment. Utilizing the Powershell deployment capability for demos from the platform training kit, my partner in crime James Prompanya and I have been testing a Powershell-only solution. For those who just want to deploy a cluster without having to open a Dev tool, you are in luck!
PS C:\Users\user\desktop\WAHSPowershellDeployment\Code> .\Deploy-HPCWindowsAzureScheduler.ps1 -InputCSCfg
.\ServiceConfigurationSample.cscfg -InputCSDef .\ServiceDefinitionSample.csdef //1. Running the script
cmdlet Deploy-HPCWindowsAzureScheduler.ps1 at command pipeline position 1
Supply values for the following parameters:
SubscriptionId: fe5717d5-c56d-4fc2-b408-b7************ //2. paste in your subscription ID from your acct.
C:\Program Files\Windows Azure HPC Scheduler SDK\v1.6\
Select the Windows Azure management certificate
0: Generate a new self-signed certificate
…………….
3: CN=HPC Test Azure Management (C4361EE9AE8FAFE44D337AF104B******)
…………….
More information: Overview of Certificates in Windows Azure <https://msdn.microsoft.com/en-us/library/gg981935.aspx>
3 //3. select your management certificate or generate one.
Select the hosted service
0: Create a new one
1: androidazure
2: hpcido2
…….
6: imgrndr
7: paraview
8: showazure
Note: Any existing deployments on the selected service will be deleted
8 // 4. Select hosted service, or goto Azure portal to create a hosted service.
Certificate CN=showazure.cloudapp.net for hosted service showazure found.
Do you wish to use this certificate for password encryption and SSL (recommended)?
[Y] Yes [A] Yes to All [N] No [L] No to All [S] Suspend [?] Help
(default is "Y"):
Select the storage account to use
0: Create a new one
2: hpcido2
3: hpcido9
4: hpcwenming
Note: You can reuse storage accounts for multiple deployments.
4 // 5. Select the storage account you want to use.
Choose the SQL Azure server to use
0: Create a new one
…………….
3: u9qtlk26j7 (Login: hpcadmin)
4: yzyp25dujg (Login: hpcadmin)
Note: You can reuse SQL Azure servers that have multiple databases.
3 //6. Select a Database which the HPC scheduler relies on.
Type the account information for the SQL Azure (u9qtlk26j7 Login: hpcadmin)
Type the SQL Azure server administrator password: ********
Choose the database to use
0: Create a new one
Note: You can only use one database per deployment
0
Specify a name for your new database: hpcscheduler // 7. Database setup
Gathering information related to your Windows Azure HPC Scheduler cluster
Specify a value for the Windows Azure HPC Scheduler administrator user name: admin
Specify a value for the Windows Azure HPC Scheduler administrator password: ********
Specify a value for the Windows Azure HPC Scheduler administrator password (again): ********
Packaging Windows Azure HPC Scheduler
Windows(R) Azure(TM) Packaging Tool version 1.6.0.0
for Microsoft(R) .NET Framework 3.5
Copyright (c) Microsoft Corporation. All rights reserved.
Examine the DB existence...
The HPC tables do not exist.
Initializing DB schema...
Uploading SchedulerSchema.sql
Uploading SchedulerSchemaAzure.sql
Uploading SchedulerSP.sql
Uploading SchedulerSPAzure.sql
Uploading CreateHPCDISCDatabase.sql
Uploading CreateHPCDISCStoredProcedures.sql
Successfully installed schema. Elapsed time: 144.9014887
Adding node HeadNodeRole1 with 2 cores on worker role HeadNodeRole, initially offline
Adding node NodeRole1 with 2 cores on worker role NodeRole, initially online
Adding node NodeRole2 with 2 cores on worker role NodeRole, initially online
Adding node NodeRole3 with 2 cores on worker role NodeRole, initially online
Setup the storage tables...
Finished.
Deploying (cspkg:C:\Users\user\desktop\WAHSPowershellDeployment\Code\Windows
AzureSchedulerDeployment\Package\output.cspkg, cscfg:C:\Users\user\desktop\W
AHSPowershellDeployment\Code\WindowsAzureSchedulerDeployment\Package\output.cscfg...
Upload Completed. (Time: 1.73307329mins Speed: 996.977821597685kb/s)
Deployment is ready!
Deployment to hosted service showazure completed
PS C:\Users\user\desktop\WAHSPowershellDeployment\Code>
I’ll discuss application installation in the Next blog. We would love to get your feedback on this, thank you. https://dl.dropbox.com/u/5859/New%20folder/WAHSPowershellDeployment.zip
Comments
Anonymous
December 07, 2011
Really useful. Your average sys. admin. hates visual studio and loves powershell, so this is a great little tool. Thanks.Anonymous
January 17, 2012
In this deployment I created new certificate. After deployment, I cannot access azure web scheduler because of error 403 which is 403 - Forbidden: Access is denied. You do not have permission to view this directory or page using the credentials that you supplied. How can I sove this problem? Do have any idea?Anonymous
January 24, 2012
Insik: the package has been updated, please try again.