Using CloudBlitz to Submit Jobs
In the second post about Cloudblitz (https://cloudblitz.codeplex.com) we'll examine how to use it to submit jobs to a deployed Azure cluster.
Keep in mind that in Microsoft's HPC scheduler implementation, jobs are just containers. They contain one or more tasks to be executed and have properties associated to them (e.g. credentials, priority).
Such a container can be created empty, then filled with tasks and finally submitted for execution. If you have a look at the Create-InstallJob.ps1 script that comes with Cloudblitz, you'll see an example of that.
Let us suppose that you want to write a simple job submission script:
1. Import the Cloudblitz.Powershell module and get the hpc object types:
Import-MyModule -Name "CloudBlitz.Powershell" -Path $cmdletsPath
$hpcTypesPath = [System.IO.Path]::GetDirectoryName($cmdletsPath) + "\HpcSchedulerManagement.dll"
Add-Type -Path $hpcTypesPath -ErrorAction Stop
2. Provide the credentials for the jobs to be submitted
# Fill in hpc scheduler credentials
$hpcCredentials = New-Object CloudBlitz.Powershell.Cmdlets.HpcSchedulerCredentials
$hpcCredentials.ClusterName = $ClusterName
$hpcCredentials.AdminLogin = $AdminLogin
$hpcCredentials.AdminPassword = $AdminPassword
$hpcCredentials.CertificateThumbprint = $mgmtcert.Thumbprint
Note that you will need the thumbprint of the certificate associated with the service at deployment. That is required because the powershell commandlets use the hpc scheduler REST API to communicate with the service. All the parameters can be passed on the command line or retrieved from the ConfigurationVariables.ps1 file that you edited before deployment.
3. Create an empty job container
# Create the job on the scheduler
$Job = New-Object HpcSchedulerManagement.DataContracts.SchedulerJob
$Job.FailOnTaskFailure = $True
$Job.Name = "Job"+$(get-random 1000000)
$Job.Id = Add-SchedulerJob -Credentials $hpcCredentials -Job $Job -ErrorAction Stop
The first line creates an empty container, then we set some properties (e.g. random name) and finally we create the job on the hpc scheduler and retrieve its id.
4. Add tasks into the job container
# Create the task
$Task = New-Object HpcSchedulerManagement.DataContracts.SchedulerTask
$Task.Name = "Task"+$(Get-Random 1000000)
$Task.CommandLine = $CommandLine
$Task.Id = Add-SchedulerTask -Credentials $hpcCredentials -JobId $Job.Id -Task $Task
As with the job, we create an empty object, set its properties, then add it to the job container whose id we retrieved before. Note that one of those properties is $Task.CommandLine, which takes a string. The string is an arbitrary command to be executed on the remote nodes. If you want to start a MPI job, it will look like
mpiexec <mpi parameters> <your mpi>.exe <app parameters>
5. Submit the job for execution
# Submit the job
Write-Host "Submitting the job"
Submit-SchedulerJob -Credentials $hpcCredentials -JobId $Job.Id
# Wait for the job to complete
Write-Host "Waiting for job $($Job.Id) to complete"
$finalState = Wait-SchedulerJob -Credentials $hpcCredentials -JobId $Job.Id
Write-Host "Job $finalState"
Note that you need not wait for the job to complete. Those lines just show a way to retrieve and display the final job state. Normally, you will submit, then query for status later with something like:
Get-schedulerjob -Credentials $hpcCredentials -JobId $JobId
That line will retrieve the job object and also display all its properties (not just those set in these samples).
You can retrieve the whole list of jobs in the scheduler with:
Get-schedulerjobList -Credentials $hpcCredentials
The job properties are those supported by Microsoft's HPC scheduler (https://msdn.microsoft.com/en-us/library/microsoft.hpc.scheduler.ischedulerjob_properties(v=vs.85).aspx).
Let's see an example of those, retrieved using Get-SchedulerJob:d : 8
Name : Job641676
UserName : topguy
Project :
RuntimeSeconds : 0
MinCores : 1
MaxCores : 1
MinSockets : 1
MaxSockets : 1
MinNodes : 1
MaxNodes : 1
UnitType : Core
RequestedNodes :
IsExclusive : False
RunUntilCanceled : False
NodeGroups :
FailOnTaskFailure : True
AutoCalculateMax : True
AutoCalculateMin : True
Preemptable : True
MinMemory : 0
MaxMemory : 0
MinCoresPerNode : 0
MaxCoresPerNode : 0
SoftwareLicense :
OrderBy :
ClientSource : AzureRestServiceAgent
Progress : 100
ProgressMessage :
TargetResourceCount : 0
ExpandedPriority : 2000
ServiceName :
NotifyOnStart : False
NotifyOnCompletion : False
EmailAddress :
Priority : Normal
Password :
JobTemplate : Default
JobType :
RequeueCount : 0
AutoRequeueCount :
RequestCancel :
Owner : topguy
SubmitTime : 5/15/2013 11:20:47 PM
CreateTime : 5/15/2013 11:20:46 PM
StartTime : 5/15/2013 11:20:48 PM
EndTime : 5/15/2013 11:20:50 PM
State : Finished
HoldUntil :
HasRuntime : False
CanGrow : True
CanShrink : True
PreviousState : Running
ChangeTime : 5/15/2013 11:20:50 PM
ExcludedNodes :
AllocatedNodes :
You will typically want to set at job submission MinCores and MaxCores (min. and max n. of cores required, default is 1). Note that properties are inherited by the tasks, unless overridden by those set at the task level (https://msdn.microsoft.com/en-us/library/microsoft.hpc.scheduler.ischedulertask_properties(v=vs.85).aspx). For MPI jobs the number of cores is fixed at submission, so MinCores=MaxCores.