次の方法で共有

Making a Clean Sweep with Windows HPC Server 2008

  • [アーティクル]

Parameter Sweeps are one of the most common types of jobs that get run on HPC clusters, and we've done some work in Windows HPC Server 2008 to make them easier (and faster) than ever. Today I'll dig in a little on what's different and how to take advantage of these new features.

For those familiar with other scheduling products, you may recognize these as very similar to "Job Arrays."

 

What is a Parameter Sweep?

Simply put, a parameter sweep is when you run a single command many times over a set of different input parameters. Problems which can be solved in this way are very common. They're also a great use of HPC clusters, since parameter sweeps are inherently embarrassingly parallel; namely they can be run in parallel with little or no effort and scale almost limitlessly.

In general, Parameter Sweeps take the form of a single command line which is run N times, with different input, output, and/or command line arguments for each of the N steps. For example, think about an application called FileZipper.exe that takes an input file and generates a compressed output file. To run it on 100 data files, you'd basically want to do something like:

FileZipper.exe <FileToZip1.dat >ZippedFile1.zip

FileZipper.exe <FileToZip2.dat >ZippedFile2.zip

FileZipper.exe <FileToZip3.dat >ZippedFile3.zip

…

FileZipper.exe <FileToZip100.dat >ZippedFile100.zip

These instances can already independently of one another, and you can run as many in parallel as you have processors to do the compression with.

Make sense? If it does, then you already understand pretty much everything there is to know about Parameter Sweeps.

 

What's different in Windows HPC Server 2008?

In the Compute Cluster Pack (our product from 2005), parameter sweeps could be generated pretty easily from the UI by inputting the Start Index (the number to start counting from), End Index (the last number to use use), and the Increment (the number to add for each step in the sweep). Using the UI to create such a sweep would then create N individual tasks in the job. This was useful, but had a number of downsides, namely:

  • You're storing a lot of repeated information in the scheduler database
  • If you wanted to change a part of your sweep, you had to change every task
  • Large sweeps became very unwieldy to view in the UI or at the command line

In Windows HPC Server 2008, we add a new type of task called a Parametric Task:

 

Figure 1: Adding a Parametric Task to a Job

Now, these tasks are stored as a unit, which makes, storing, editing, managing, and monitoring them a snap! Let's go ahead and give it a try . . .

 

Creating a Parameter Sweep

To create your first parameter sweep, open up the HPC Job Manager. The simplest way to create a sweep is to click on the Parametric Sweep Job link in the right-hand Actions Pane.

Let's create a sweep in the form of the example up above; namely, a sweep that zips up 100 files:

1. Go ahead and provide a Name like "File Zipper".

2. Set the Start Value of 1 and an End Value of 2.

3. Leave the Increment Value to be 1 (since we'll be counting up by 1's).

4. Enter your Command Line, in this case "FileZipper.exe".

5. We'll see the Standard Input and Standard Output as above:

1. Stdin: FileToZip*.dat

2. Stdout: ZippedFile*.zip

6. Check the preview box at the bottom of the dialog to see what your sweep will look like, then go ahead and submit.

You should end up with something that looks like this:

Figure 2: Creating a Parameter Sweep Task

 

Tracking Your Sweep's Progress

If you check the job list, you should now see that you've submitted a job with a single task in it. But actually, you can easily track each task individually by checking the box labeled Expand parametric tasks. This allows you track your sweep as a unit, but dig in on failures or results for individual steps.

 

Doing that From the Command Line

Of course you can do the same thing from the Command Line:

C:\>job submit /parametric:100 /StdIn:"FileToZip*.dat" /StdOut:"ZippedFile*.zip" FileZipper.exe

Or from PowerShell:

PS> New-HpcJob | Add-HpcTask -Parametric -Start 1 -End 100 -Stdin "FileToZip*.dat" -Stdout "ZippedFile*.zip" -CommandLine "FileZipper.exe" | Submit-HpcJob

 

That's all for this time. Happy sweeping!