Understanding Backfilling
Backfilling can help maximize cluster utilization and throughput by allowing smaller jobs farther back in the queue to run ahead of a job waiting at the front of the queue, as long as the job at the front is not delayed as a result. The HPC Job Scheduler Service grows running jobs to their maximum requested resources before it tries to utilize resources for backfilling. For information about how to change the configuration options, see Configure the HPC Job Scheduler Service.
Backfilling is enabled by default, with the backfill look ahead set at 1000 jobs. The backfill look ahead setting determines the number of jobs that the HPC Job Scheduler service searches to find jobs that can fill gaps in the schedule. The following configuration options allow you to modify or disable backfilling:
Allow backfilling from part of the queue (within the specified backfill look ahead)
Allow backfilling from the entire queue
Do not allow backfilling
Note
Backfilling is only effective when jobs submitted to the cluster have a maximum run time specified.
You can use job templates to define a maximum run time on all jobs. For example, you can create a series of job templates named BigJob, MediumJob, and SmallJob with maximum run times of one day, one hour, and one minute, respectively. For more information, see Job Templates.
You can also write a job submission filter that checks that the runtime job property is not set to infinite. For more information, see Understanding Activation and Submission Filters.
How backfilling works
When a job reaches the top of the queue, a sufficient number of nodes may not be available to meet its minimum core requirement. When this happens, the job reserves any nodes that are immediately available and waits for the job that is currently running to complete.
Backfilling then utilizes the reserved idle nodes as follows:
Based on the run time specified for the job that is currently running, a start time for the waiting job is established.
The start time is used to define a backfill window of nodes (n) x time (t). For example, four nodes that idle for 15 minutes would create a 4 x 15 backfill window.
The HPC Job Scheduler Service searches for the first job in the queue that can complete within the backfill window. For example, a job that requires a minimum of eight cores (four nodes, assuming dual-core nodes) and has a run time of 10 minutes would fit the 4 x 15 window exactly.
If a job is found that fits the window, it is activated and run ahead of the job that is waiting at the top of the queue.