MedianStoppingPolicy Class

Defines an early termination policy based on running averages of the primary metric of all runs.

Initialize a MedianStoppingPolicy.

Inheritance

azureml.train.hyperdrive.policy.EarlyTerminationPolicy

MedianStoppingPolicy

Constructor

MedianStoppingPolicy(evaluation_interval=1, delay_evaluation=0)

Parameters

Name	Description
evaluation_interval	int The frequency for applying the policy. Default value: 1
delay_evaluation	int The number of intervals for which to delay the first policy evaluation. If specified, the policy applies every multiple of evaluation_interval that is greater than or equal to delay_evaluation. Default value: 0
evaluation_interval Required	int The frequency for applying the policy.
delay_evaluation Required	int The number of intervals for which to delay the first policy evaluation. If specified, the policy applies every multiple of evaluation_interval that is greater than or equal to delay_evaluation.

Remarks

The Median Stopping policy computes running averages across all runs and cancels runs whose best performance is worse than the median of the running averages. Specifically, a run will be canceled at interval N if its best primary metric reported up to interval N is worse than the median of the running averages for intervals 1:N across all runs.

The Median Stopping policy takes the following optional configuration parameters:

• evaluation_interval: The frequency for applying the policy. Each time the training script logs the primary metric counts as one interval.

• delay_evaluation: The number of intervals to delay policy evaluation. Use this parameter to avoid premature termination of training runs. If specified, the policy applies every multiple of evaluation_interval that is greater than or equal to delay_evaluation.

This policy is inspired from the research publication Google Vizier: A Service for Black-Box Optimization.

If you are looking for a conservative policy that provides savings without terminating promising jobs, you can use a Median Stopping Policy with evaluation_interval 1 and delay_evaluation 5. These are conservative settings, that can provide approximately 25%-35% savings with no loss on primary metric (based on our evaluation data).

Attributes

delay_evaluation

Return the value for the number of sequences the first evaluation is delayed.

Returns

Type	Description
int	The delay evaluation.

evaluation_interval

Return evaluation interval value.

Returns

Type	Description
int	The evaluation interval.

POLICY_NAME

POLICY_NAME = 'MedianStopping'