Sessions that are running on Azure nodes are more likely to reach the message resend limits than sessions that are running on on-premises nodes, especially if the HPC Job Scheduler Service is running in Balanced mode. If you see messages fail on your Azure nodes, try increasing the message resend limit (the default value is 3). This attribute can be set in the load balancing settings in the service configuration file (the <loadBalancing> element is in the <Microsoft.Hpc.Broker> section). For more information, see the broker settings section in SOA Service Configuration Files in Windows HPC Server 2008 R2.
Upload a SOA service to a Windows Azure storage account
Scenario
You want to deploy or have deployed a set of Windows Azure nodes in your HPC cluster, and you want to upload SOA service files to the Windows Azure storage account.
Note
When you provision a set of Windows Azure nodes from HPC Cluster Manager, any applications or files that are on the storage account are automatically deployed to the Azure nodes. If you upload file packages to storage after the Azure nodes are started, you can use clusrun and HpcSync to manually deploy the files to the Azure nodes. For more information, see Manually deploy uploaded packages to Windows Azure nodes.
Goal
Create a package and upload SOA service files to a Windows Azure storage account.
Requirements
A head node with Windows HPC Server 2008 R2 SP 1 installed.
A Windows Azure subscription.
An Azure Worker Role node template created.
Important considerations
Azure nodes cannot access on-premises nodes or shares directly.
Services that get data through message requests can run on Azure with no changes to the service or the client. Services that require access to databases or other external data sources must include code that uses the Azure APIs to access data.
Tasks executing on an Azure node cannot instantiate a connection with the head node.
To submit SOA jobs to the cluster, you must have a WCF Broker Node configured and Online.
Steps
To package and upload your SOA service files to the Windows Azure storage account:
If the SOA service is not already registered and deployed to the on-premises cluster, register the SOA service by placing a copy of the service configuration file in the service registration folder on the head node (typically this is C:\Program Files\Microsoft HPC Pack 2008 R2\ServiceRegistration). For detailed information, see Deploy and Edit the Service Configuration File.
Copy the service configuration file, the service assembly, and any dependent DLLs to an empty folder. For example, to a folder named C:\AzurePackages\myServiceFiles.
At a command prompt window, run HpcPack create and specify a name for your package and specify the folder that contains the files that you want to package.
Important
The name of the package must be the name of the SOA service (that is, the service name that the SOA client specifies in the SessionStartInfo constructor).
For example, to package the content of C:\AzurePackages\myServiceFiles as myServiceName.zip:
Run hpcPack upload to upload the package to your Windows Azure storage account. You can specify an account by using the node template name, and optionally the name of the head node (if there is no default head node specified on your computer). For example:
Run HpcPack list to verify that the package was uploaded. For example:
Hpcpack list /nodetemplate:myAzureNodeTemplate /scheduler:myHeadNode
If your Azure nodes are already started, then you must deploy the uploaded packages before you can run your SOA jobs. See Manually deploy uploaded packages to Windows Azure nodes.
Expected results
You can run the service client with no changes, just as you would if the service were running on-premises.
When you run hpcPack list, you see information about the package that you uploaded.
Upload an XLL file to a Windows Azure storage account
Scenario
You want to deploy or have deployed a set of Windows Azure nodes in your HPC cluster, and you want to upload XLL files to the Windows Azure storage account so that you can run your UDF offloading jobs on Azure nodes.
Note
When you provision a set of Windows Azure nodes from HPC Cluster Manager, any applications or files that are on the storage account are automatically deployed to the Azure nodes. If you upload file packages to storage after the Azure nodes are started, you can use clusrun and HpcSync to manually deploy the files to the Azure nodes. For more information, see Manually deploy uploaded packages to Windows Azure nodes.
Goal
Create a package and upload an XLL file to a Windows Azure storage account.
Requirements
A head node with Windows HPC Server 2008 R2 SP 1 installed.
A Windows Azure subscription.
An Azure Worker Role node template created.
A cluster-enabled XLL file.
Important considerations
Azure nodes cannot access on-premises nodes or shares directly.
To submit UDF offloading jobs to the cluster, you must have a WCF Broker Node configured and Online.
Steps
To package and upload your XLL files to the Windows Azure storage account:
At a command prompt window, run HpcPack create to package your XLL. The name of the package must be the same as the name of the XLL file.
For example, to package C:\myFiles\myXLL.xll as myXLL.zip (and save the package to a folder called AzurePackages):
If the XLL has dependencies on DLLs or other files, copy the XLL and its dependencies to a folder, and specify the name of the folder instead of the .xll file in the hpcPack create command. For example: hpcpack create C:\AzurePackages\myXLL.zip C:\myFiles\myXLLFolder
Run hpcPack upload to upload the package to your Windows Azure storage account. You can specify an account by using the node template name, and optionally the name of the head node (if there is no default head node specified on your computer). For example:
Run HpcPack list to verify that the package was uploaded. For example:
Hpcpack list /nodetemplate:myAzureNodeTemplate /scheduler:myHeadNode
If your Azure nodes are already started, then you must deploy the uploaded packages before you can run your UDF jobs. See Manually deploy uploaded packages to Windows Azure nodes.
Expected results
You can offload UDFs to the cluster from Excel 2010 with no changes, just as you would if the XLLs were deployed on-premises.
When you run hpcPack list, you see information about the package that you uploaded.
Upload batch job assemblies to a Windows Azure storage account
Scenario
You want to deploy or have deployed a set of Windows Azure nodes in your HPC cluster, and you want to upload batch job assembly files to the Windows Azure storage account.
Note
When you provision a set of Windows Azure nodes from HPC Cluster Manager, any applications or files that were copied to the storage account using hpcPack upload are automatically deployed to the Azure nodes. If you upload file packages to storage after the Azure nodes are started, you can use clusrun and HpcSync to manually deploy the files to the Azure nodes. For more information, see Manually deploy uploaded packages to Windows Azure nodes.
Goal
Create a package and upload files to a Windows Azure storage account.
Requirements
A head node with Windows HPC Server 2008 R2 SP 1 installed.
A Windows Azure subscription.
An Azure Worker Role node template created.
Important considerations
Azure nodes cannot access on-premises nodes or shares directly. For example, an executable in a task cannot write data to a file on a share or redirect data to a file on a share. You can package input data with your executable and upload it to the worker nodes. Task output up to 4MB can go to the job scheduler database (accessible in the task’s output field).
Tasks executing on an Azure node cannot instantiate a connection with the head node. For example, a task cannot contact the job scheduler to get information about the job, and a task cannot spawn a new task or job.
Steps
To package and upload your files to the Windows Azure storage account:
Create a folder on your head node for your packages, such as C:\AzurePackages.
At a command prompt window, run HpcPack create and specify a name for your package and specify and the folder or files that you want to package. For example:
Run hpcPack upload to upload the package to your Windows Azure storage account. You can specify an account by using the node template name, and optionally the name of the head node (if there is no default head node specified on your computer). For example:
When you specify a relative path, you can use that in your command line when you submit the batch job. For example, job submit %CCP_PACKAGE_ROOT%\<relativePath>\myExe.exe.
Run HpcPack list to verify that the package was uploaded. For example:
Hpcpack list /nodetemplate:myAzureNodeTemplate /scheduler:myHeadNode
If your Azure nodes are already started, then you must deploy the uploaded packages before you can run your batch jobs. See Manually deploy uploaded packages to Windows Azure nodes.
Expected results
You can run a batch job on Azure Worker Role nodes in the same way that you run jobs on on-premises nodes. For example, job submit %CCP_PACKAGE_ROOT%\<relativePath>\myExe.exe.
When you run hpcPack list, you see information about the package that you uploaded.
Manually deploy uploaded packages to Windows Azure nodes
Scenario
You have uploaded new batch assembly, XLL, or service file packages to the Azure storage account associated with a set of running Windows Azure nodes. You want to deploy the new files to the worker nodes.
Note
The worker nodes automatically run HpcSync when they are restarted. HpcSync copies all packages from the storage account to the worker nodes.
Goal
Deploy packages to the worker nodes.
Requirements
A head node with Windows HPC Server 2008 R2 SP1 installed.
A set of Windows Azure nodes joined to your HPC cluster.
One or more packages deployed to your Windows Azure storage account.
Steps
You can use clusrun and HpcSync to deploy the files from the Windows Azure storage account to the Windows Azure nodes.
For example:
clusrun /nodegroup:AzureWorkerNodes hpcsync
To see a list of folders or files that have been deployed to the Windows Azure nodes, you can run the following command:
clusrun /nodegroup:AzureWorkerNodes dir %CCP_PACKAGE_ROOT% /s
Expected results
The files are successfully deployed to the worker nodes.
The scenarios in this section help you try new management features in HPC Pack 2008 R2 SP 1.
Configure availability of workstation nodes based on detected user activity
Scenario
A portion of your HPC workload consists of small, short-running jobs that can be stopped and re-started. These are ideal jobs to run on workstation nodes. You want to use the workstations on your network during working and non-working hours, but only if the workstation users are not actively using them. You don’t want the workstation to come online for HPC jobs if user activity is detected, and you want HPC jobs to immediately vacate the workstation when users become active again.
Goal
Add workstation nodes to your HPC cluster and configure the availability policy so that nodes come online when there is no user activity detected. In the availability policy, define the following:
The days and times during which you want to try to use workstations for HPC jobs.
The number of minutes without keyboard or mouse input after which a workstation is considered idle.
The CPU usage threshold (that is, usage must be under the threshold for the workstation to be considered idle).
Requirements
A cluster with HPC Pack 2008 R2 SP1 installed.
One or more workstation computers running the Windows 7 operating system.
The workstation computers and the head node computer must be joined to the same doMayn.
Administrative permissions on the cluster.
Steps
To configure availability of workstation nodes based on detected user activity:
In HPC Cluster Manager, create a node template for the workstations:
In Configuration, click Node Templates, and then click New.
Use the Create Node Template Wizard to create a Workstation node template.
On the Configure Availability Policy page, select Bring workstation nodes online and offline automatically and then click Configure Availability Policy.
In the availability policy dialog box, select the days and times during which you want to try to use the workstations for HPC jobs.
In the user activity detection options:
Select the checkbox and configure the number of minutes without keyboard or mouse input after which the workstation can be brought online for jobs.
Select the checkbox and configure the threshold for CPU usage. For example, if you select 30%, a workstation with a CPU usage over 30% will not be brought online for jobs. (This option is only available if the keyboard input option is selected.)
Install HPC Pack 2008 R2 SP1 on the workstations and select the Join an existing HPC cluster by creating a new workstation node option.
The nodes appear in Node Management as Unapproved. If the nodes do not appear, verify the network connection between the head node and the workstations.
Assign the workstation template that you created to the nodes. The workstation node state should change from Unapproved to Offline. If you have an availability policy set, then the workstation will be brought online according to the policy.
Important
If the workstation nodes have HPC Pack 2008 R2 but do not have SP1 installed, the activity detection settings in the node template cannot be applied. The workstations will be brought online according to the days and times that you selected in the node template.
If you have some workstation nodes with only HPC Pack 2008 R2 (version 3.0.xxxx.x) and some with SP1 installed (version 3.1.xxxx.x), you can create separate node templates with different availability policies. In HPC Cluster Manager, you can add the Version column in the node list view, and then sort the nodes by version number.
Expected results
Workstation nodes become available according to the configured availability policy and user activity detection options.
HPC workload exits quickly when a workstation becomes active (no noticeable delay for the workstation user).
Workstation nodes that do not have SP1 installed become available only according to the days and times that you selected in the node template. The user activity detection settings are not applied.
The scenarios in this section help you try new job scheduling features in HPC Pack 2008 R2 SP 1.
Mark a task as critical
Scenario
One or more specific tasks in your job are critical. If those tasks fail, you want the entire job to stop running and be marked as Failed.
Alternately, you might be running a SOA job (or other job with sub-tasks), and it is okay if a few instances of the service fail (this can happen if a particular sub-task is pre-empted or stopped while running for any reason). However, if enough sub-tasks fail, this could indicate a problem with the service itself, so you want the SOA job to stop and be marked as Failed.
Goal
Create a task with the failJobOnFailure task property set to true. This marks the task as critical, and causes the job to stop and be marked as Failed if the task fails.
Create a task with the FailJobOnFailureCount task property set to specify how many sub-tasks can fail before the task reaches a critical failure level and causes the job to stop and be marked as Failed.
Note
When a job is failed due to the failure of a critical task, the tasks in the job will be canceled (running tasks will be marked as Failed, queued tasks will reMayn marked as Queued), and the job will be marked as Failed.
Requirements
A cluster with HPC Pack 2008 R2 SP1 installed.
The SP1 client utilities installed on the client computer.
User permissions on the cluster.
Steps
To mark a task as critical, you can set the failJobOnFailure task property to true. For Parametric Sweep and Service tasks, you can additionally set the FailJobOnFailureCount to specify how many sub-tasks can fail before the job should be stopped and failed.
For example, you can use one of the following methods to mark a task as critical:
You want to build a dependency tree of jobs where a task in one job configures and submits new jobs.
Goal
Submit a job that configures and submits new jobs.
Requirements
A cluster with HPC Pack 2008 R2 SP 1 installed.
The SP1 client utilities installed on the client computer.
User permissions on the cluster.
Credential reuse enabled on the cluster.
Note
HPC Pack 2008 R2 SP1 includes a cluster property named DisableCredentialReuse. By default, this is set to false (that is, credential reuse is enabled).
Steps
In previous releases, to enable a job to submit new jobs, you had to either set your credentials on all compute nodes or provide your password through the task that submitted the new jobs. In this release, if you select to save your password (or set credentials with cluscfg setcreds), your credentials can be reused by your job.
To submit a job that submits new jobs:
Create and submit a job that configures and submits new jobs.
For example, to submit a parametric sweep that creates and submits four new jobs, type the following at a command prompt window:
.gif)
Note.gif)
Important