Manage Azure Data Lake Analytics using Account Policies
Account policies help you control how resources an Azure Data Lake Analytics account are used. These policies allow you to control the cost of using Azure Data Lake Analytics. For example, with these policies you can prevent unexpected cost spikes by limiting how many AUs the account can simultaneously use.
Important
Azure Data Lake Analytics retired on 29 February 2024. Learn more with this announcement.
For data analytics, your organization can use Azure Synapse Analytics or Microsoft Fabric.
Account-level policies
These policies apply to all jobs in a Data Lake Analytics account.
Maximum number of AUs in a Data Lake Analytics account
A policy controls the total number of Analytics Units (AUs) your Data Lake Analytics account can use. By default, the value is set to 250. For example, if this value is set to 250 AUs, you can have one job running with 250 AUs assigned to it, or 10 jobs running with 25 AUs each. Other jobs that are submitted are queued until the running jobs are finished. When running jobs are finished, AUs are freed up for the queued jobs to run.
To change the number of AUs for your Data Lake Analytics account:
In the Azure portal, go to your Data Lake Analytics account.
Select Limits and policies.
Under Maximum AUs, move the slider to select a value, or enter the value in the text box.
Select Save.
Note
If you need more than the default (250) AUs, in the portal, click Help+Support to submit a support request. The number of AUs available in your Data Lake Analytics account can be increased.
Maximum number of jobs that can run simultaneously
This policy limits how many jobs can run simultaneously. By default, this value is set to 20. If your Data Lake Analytics has AUs available, new jobs are scheduled to run immediately until the total number of running jobs reaches the value of this policy. When you reach the maximum number of jobs that can run simultaneously, subsequent jobs are queued in priority order until one or more running jobs complete (depending on available AUs).
To change the number of jobs that can run simultaneously:
In the Azure portal, go to your Data Lake Analytics account.
Select Limits and policies.
Under Maximum Number of Running Jobs, move the slider to select a value, or enter the value in the text box.
Select Save.
Note
If you need to run more than the default (20) number of jobs, in the portal, click Help+Support to submit a support request. The number of jobs that can run simultaneously in your Data Lake Analytics account can be increased.
How long to keep job metadata and resources
When your users run U-SQL jobs, the Data Lake Analytics service keeps all related files. These files include the U-SQL script, the DLL files referenced in the U-SQL script, compiled resources, and statistics. The files are in the /system/ folder of the default Azure Data Lake Storage account. This policy controls how long these resources are stored before they're automatically deleted (the default is 30 days). You can use these files for debugging, and for performance-tuning of jobs that you'll rerun in the future.
To change how long to keep job metadata and resources:
- In the Azure portal, go to your Data Lake Analytics account.
- Select Limits and policies.
- Under Days to Retain Job Queries, move the slider to select a value, or enter the value in the text box.
- Select Save.
Job-level policies
Job-level policies allow you to control the maximum AUs and the maximum priority that individual users (or members of specific security groups) can set on jobs that they submit. This policy lets you control the costs incurred by users. It also lets you control the effect that scheduled jobs might have on high-priority production jobs that are running in the same Data Lake Analytics account.
Data Lake Analytics has two policies that you can set at the job level:
AU limit per job: Users can only submit jobs that have up to this number of AUs. By default, this limit is the same as the maximum AU limit for the account.
Priority: Users can only submit jobs that have a priority lower than or equal to this value. A higher number indicates a lower priority. By default, this limit is set to 1, which is the highest possible priority.
There's a default policy set on every account. The default policy applies to all users of the account. You can create more policies for specific users and groups.
Note
Account-level policies and job-level policies apply simultaneously.
Add a policy for a specific user or group
In the Azure portal, go to your Data Lake Analytics account.
Select Limits and policies.
Under Job Submission Limits, select the Add Policy button. Then, select or enter the following settings:
Compute Policy Name: Enter a policy name, to remind you of the purpose of the policy.
Select User or Group: Select the user or group this policy applies to.
Set the Job AU Limit: Set the AU limit that applies to the selected user or group.
Set the Priority Limit: Set the priority limit that applies to the selected user or group.
Select Ok.
The new policy is listed in the Default policy table, under Job Submission Limits.
Delete or edit an existing policy
In the Azure portal, go to your Data Lake Analytics account.
Select Limits and policies.
Under Job Submission Limits, find the policy you want to edit.
To see the Delete and Edit options, in the rightmost column of the table, select
....