Monitor your usage of Unity Catalog resource quotas

Άρθρο
09/25/2024

This article describes how to monitor your usage of Unity Catalog securable objects that are subject to resource quotas.

You can use the Unity Catalog resource quotas APIs to track usage. Although some limits can be increased upon request, others are fixed. To avoid disruptions, plan ahead and contact your Azure Databricks account team if you anticipate exceeding your resource quotas.

What are Unity Catalog resource quotas?

Unity Catalog enforces resource quotas on all securable objects managed by Unity Catalog. These quotas are listed in Resource limits. They are identified in that article as quotas for Clean Rooms, Delta Sharing, Marketplace, and Unity Catalog.

Each quota is defined as a number of objects per parent object (or scope). For example, 10,000 tables per schema or 1,000,000 tables per metastore.

Query your usage against resource quotas

To monitor usage against resource quotas proactively, use the Unity Catalog Resource Quotas REST APIs:

GetQuota retrieves quota usage for one quota type, defined as the number of child objects per parent (for example, tables per metastore).
ListQuotasretrieves all quota values under the target metastore, paginated by default.

Both APIs return information in the form of a quota_info object that contains the following fields. You also use some of these fields when you make a request using the GetQuota API:

parent_securable_type: Type of the parent object. For example, for the tables-per-schema count, the parent_securable_type is schema.

Note

For quotas in which the parent type is a registered model, set the parent_securable_type to function.
parent_full_name: Full name of the quota parent. For example, the main.default schema. If the parent is a metastore, use the metastore ID in your request.
quota_name: Name of the quota. This is the child object (table, schema, share, and so forth) suffixed by -quota. For example, table-quota.
quota_count: The latest usage count. For example, 33 tables per schema.
quota_limit: The quota limit value at the time the quota count was calculated. For example, 10000 tables per schema.
last_refreshed_at: The last time the quota count was refreshed. This is displayed as a Unix epoch timestamp. You can convert the timestamp to a human-readable format using online tools such as Epoch Converter.

The ListQuotas API also returns a page token in the response if the current response doesn’t return all results.

API authorization and authentication

Only account admins can call the Resource Quotas APIs.

The account admin who calls the APIs should use either OAuth user-to-machine (U2M) authentication (for users or groups) or OAuth machine-to-machine (M2M) authentication (if the account admin is a service principal). See Authenticate access to Azure Databricks with a user account using OAuth (OAuth U2M) or Authenticate access to Azure Databricks with a service principal using OAuth (OAuth M2M). Databricks-generated personal access tokens (PATs) are also an option, but not recommended.

Use the GetQuota API to get quota usage values for a specific quota type

Use the GetQuota API to get usage information for a single resource quota as defined by a child-parent pairing.

Method: GET

Path: /unity-catalog/resource-quotas/{parent_securable_type}/{parent_full_name}/{quota_name}

Body parameters: For parameter descriptions, see Query your usage against resource quotas.

For the API reference, see GET /unity-catalog/resource-quotas/.

GetQuota counts are accurate to within 30 minutes of the last create operation performed under the quota parent. The count might be out of date if only delete operations have been performed, because Unity Catalog updates the quota count only during resource creation. Calling GetQuota triggers a refresh of the quota count if it is out of date, however the trigger is asynchronous and new counts might not be returned in the first call.

Request example

Python example that requests the number of schemas that have been created in the main catalog in the metastore attached to the workspace:

import requests
headers = {'Authentication': 'Bearer <OAuthtoken>'}
r = requests.get('https://example-workspace.databricks.com/api/2.1/unity-catalog/resource-quotas/catalog/main/schema-quota', headers=headers)
print(r.text)

Curl example that does the same:

$ curl -X GET -H "Authentication: Bearer $OAUTH_TOKEN" \
"https://example-workspace.databricks.com/api/2.1/unity-catalog/resource-quotas/catalog/main/schema-quota"

Response example

Response that shows 2691 schemas against the 10,000 schema-per-metastore limit:

{
  "quota_info": {
    "parent_securable_type": "CATALOG",
    "parent_full_name": "main",
    "quota_name": "schema-quota",
    "quota_count": 2691,
    "quota_limit": 10000,
    "last_refreshed_at": 1722559381517
  }
}

Use the ListQuotas API to get usage data for all quota types in a metastore

Use the ListQuotas API to get usage data for all quota types in a metastore.

Method: GET

Path: /unity-catalog/resource-quotas/all-resource-quotas

Body parameters:

max_results: Number of results to return. Maximum value is 500. Defaults to 100.
page_token: Page token from the previous request to fetch the next page of results.

For the API reference, see GET /unity-catalog/resource-quotas/all-resource-quotas.

Unlike GetQuotas, ListQuotas has no SLA on the freshness of counts. Nor does it trigger quota count refreshes. For maximum accuracy, use the GetQuota API.

Request example

Python example that requests quota counts for all objects in the metastore attached to the workspace, specifying 5 results to return per page:

import requests
headers = {'Authentication': 'Bearer <OAuthtoken>'}
next_page = None
max_results = 5
results = []

while True:
  payload = {'max_results': max_results, 'page_token': next_page}
  r = requests.get(
'https://example-workspace.databricks.com/api/2.1/unity-catalog/resource-quotas/all-resource-quotas', headers=headers, params=payload).json()
  results.extend(r["quotas"])
  if "next_page_token" not in r: break
  next_page = r["next_page_token"]

results

Curl example that does the same:

$ curl -X GET -H "Authentication: Bearer $OAUTH_TOKEN" \
-d '{"max_results": 5}' "https://example-workspace.databricks.com/api/2.1/unity-catalog/resource-quotas/all-resource-quotas"

Response example

Response that shows one page of 5 quota counts:

"quotas":[
   {
      "parent_securable_type":"CATALOG",
      "parent_full_name":"auto_maintenance",
      "quota_name":"schema-quota",
      "quota_count":15,
      "quota_limit":10000,
      "last_refreshed_at":1707272498713
   },
   {
      "parent_securable_type":"CATALOG",
      "parent_full_name":"demo_icecream",
      "quota_name":"schema-quota",
      "quota_count":3,
      "quota_limit":10000,
      "last_refreshed_at":1720789637102
   },
   {
      "parent_securable_type":"CATALOG",
      "parent_full_name":"primarycatalog",
      "quota_name":"schema-quota",
      "quota_count":2,
      "quota_limit":10000,
      "last_refreshed_at":1720829359520
   },
   {
      "parent_securable_type":"CATALOG",
      "parent_full_name":"shared_catalog_azure",
      "quota_name":"schema-quota",
      "quota_count":670,
      "quota_limit":10000,
      "last_refreshed_at":1722036080791
   },
   {
      "parent_securable_type":"CATALOG",
      "parent_full_name":"cat-test",
      "quota_name":"schema-quota",
      "quota_count":567,
      "quota_limit":10000,
      "last_refreshed_at":1704845201239
   }
],
"next_page_token":"eyJfX3R2IjoiMCIsInB0IjoiQ2F0YWxvZyIsInBpZCI6IjAwNTAyYTM1LWIzMGQtNDc4YS1hYTIwLTE5MDZkMGVmNzdiNiIsInJ0IjoiU2NoZW1hIn0="

Κοινή χρήση μέσω

Monitor your usage of Unity Catalog resource quotas

What are Unity Catalog resource quotas?

Query your usage against resource quotas

API authorization and authentication

Use the GetQuota API to get quota usage values for a specific quota type

Request example

Response example

Use the ListQuotas API to get usage data for all quota types in a metastore

Request example

Response example

Σχόλια

Πρόσθετοι πόροι