BanditPolicy Class
Defines an early termination policy based on slack criteria, and a frequency and delay interval for evaluation.
Initialize a BanditPolicy with slack factor, slack_amount, and evaluation interval.
- Inheritance
-
azureml.train.hyperdrive.policy.EarlyTerminationPolicyBanditPolicy
Constructor
BanditPolicy(evaluation_interval=1, slack_factor=None, slack_amount=None, delay_evaluation=0)
Parameters
Name | Description |
---|---|
slack_factor
|
The ratio used to calculate the allowed distance from the best performing experiment run. Default value: None
|
slack_amount
|
The absolute distance allowed from the best performing run. Default value: None
|
evaluation_interval
|
The frequency for applying the policy. Default value: 1
|
delay_evaluation
|
The number of intervals for which to delay the first policy evaluation.
If specified, the policy applies every multiple of Default value: 0
|
slack_factor
Required
|
The ratio used to calculate the allowed distance from the best performing experiment run. |
slack_amount
Required
|
The absolute distance allowed from the best performing run. |
evaluation_interval
Required
|
The frequency for applying the policy. |
delay_evaluation
Required
|
The number of intervals for which to delay the first policy evaluation.
If specified, the policy applies every multiple of |
Remarks
The Bandit policy takes the following configuration parameters:
slack_factor
: The amount of slack allowed with respect to the best performing training run. This factor specifies the slack as a ratio.slack_amount
: The amount of slack allowed with respect to the best performing training run. This factor specifies the slack as an absolute amount.evaluation_interval
: Optional. The frequency for applying the policy. Each time the training script logs the primary metric counts as one interval.delay_evaluation
: Optional. The number of intervals to delay policy evaluation. Use this parameter to avoid premature termination of training runs. If specified, the policy applies every multiple ofevaluation_interval
that is greater than or equal todelay_evaluation
.
Any run that doesn't fall within the slack factor or slack amount of the evaluation metric with respect to the best performing run will be terminated.
Consider a Bandit policy with slack_factor
= 0.2 and evaluation_interval
= 100.
Assume that run X is the currently best performing run with an AUC (performance metric) of 0.8 after 100
intervals. Further, assume the best AUC reported for a run is Y. This policy compares the value
(Y + Y * 0.2) to 0.8, and if smaller, cancels the run. If delay_evaluation
= 200, then the
first time the policy will be applied is at interval 200.
Now, consider a Bandit policy with slack_amount
= 0.2 and evaluation_interval
= 100.
If Run 3 is the currently best performing run with an AUC (performance metric) of 0.8 after 100 intervals,
then any run with an AUC less than 0.6 (0.8 - 0.2) after 100 iterations will be terminated.
Similarly, the delay_evaluation
can also be used to delay the first termination policy
evaluation for a specific number of sequences.
For more information about applying early termination policies, see Tune hyperparameters for your model.
Attributes
delay_evaluation
Return the number of sequences for which the first evaluation is delayed.
Returns
Type | Description |
---|---|
The delay evaluation. |
evaluation_interval
slack_factor
Return the slack factor with respect to the best performing training run.
Returns
Type | Description |
---|---|
The slack factor. |
POLICY_NAME
POLICY_NAME = 'Bandit'