Share via

Managing Your Azure Data Lake Analytics Compute Resources (Job-level Policy)

  • Article

In Managing Your Azure Data Lake Analytics Compute Resources (Overview) and Account Level Policy, we gave an overview of many ways by which ADLA helps you manage your compute capacity. We also went in to the details of account level policies. In this blog, let's dig deeper in to job-level policies.

Job-level Policies

With job-level policies, you can control the maximum AUs and the maximum priority that individual users (or members of security groups) can set on the jobs that they submit. This allows you to not only control the costs incurred by your users but also control the impact they might have on high priority production jobs running in the same ADLA account.

There are two parts to a job level policy:

  • Default Policy: This is the policy that is applied to all users of the service.
  • Exceptions: The set of "exception" policies apply to specific users.

Submitted jobs that do not violate the job-level policies are still subject to the account level policies as described in Azure Data Lake Analytics Account Level Policy.

Default Policy

You can configure a default policy to control the jobs submitted by all users except those in exception policies. The default policy can control the following two limits on each job:

  • Job AU limit: Users can only submit jobs with up to this number of AUs. By default, this limit is the same as the maximum AU limit at the account level.
  • Priority limit: Users can only submit jobs with priority lower than or equal to this value. Please note that a higher number means lower priority. By default, this is set to 1 which is the highest possible priority.

If users try to submit jobs that violate the above limits, the request will be rejected with the error message, such as:“This job was rejected because it requires <XXX> AUs and this account has an administrator-defined policy that prevents a job from using more than <XXX> AUs” .

Exceptions to Policy

You can configure exception policies for certain individuals or groups of users so that they can submit jobs with different AUs or priorities than what is allowed by the default policy. You can create multiple exception policies to give different users the ability to submit jobs with different AUs and priorities. For example, refer to the following table of exception policies.

Development Group QA Group
Job AU limit 100 50
Priority limit 50 20

 

As per the two exception policies in the table above, members of the "Development Group" can submit jobs with up to 100 AUs and priority no greater than 50. Members of the "QA Group" can submit jobs with fewer compute capacity (up to 50) with priority no greater than 20.

What happens if multiple exception policies apply to the same user? If a submitter has multiple exceptions that apply to them, the most permissive limit will take effect. For example, in the table above, if Alice belongs to both the "Development Group" and the "QA Group"?, she will be able to submit jobs with 100 AUs and priority of 20. This would not have been possible if she was a member of only one of the two groups.