Share via

December 2024

  • Article

These features and Azure Databricks platform improvements were released in December 2024.

Note

Releases are staged. Your Azure Databricks account might not be updated until a week or more after the initial release date.

Databricks Runtime 16.1 is GA

December 20, 2024

Databricks Runtime 16.1 and Databricks Runtime 16.1 ML are now generally available.

See Databricks Runtime 16.1 and Databricks Runtime 16.1 for Machine Learning.

The default format for new notebooks is now IPYNB (Jupyter) format

December 20, 2024

The default format for new notebooks you create in your Azure Databricks workspace is now IPYNB (.ipynb). Previously, the default format for notebooks was Source (.py, .sql, .scala, .r). To change the default format, use the Default file format for notebooks setting in the Developer pane of your workspace user settings. See Notebook formats.

Databricks-hosted models for Databricks Assistant is in public preview

December 19, 2024

You can now use Azure Databricks-hosted models to power Databricks Assistant in Azure Databricks. This feature is in public preview.

See Use a Databricks-hosted model for Databricks Assistant.

Python code executor for AI agents (Public Preview)

December 19, 2024

You can now quickly give your AI agents the ability to run Python code. Databricks now offers a pre-built Unity Catalog function that can be used by an AI agent as a tool to expand their capabilities beyond language generation.

See Code interpreter AI agent tools.

databricks-agents SDK 0.13.0 release

December 18, 2024

Version 0.13.0 of the databricks-agents SDK has been released to PyPI, containing the following changes:

  • Honor the current active Databricks CLI profile and MLflow model registry URI when calling agents.deploy() and other databricks.agents APIs. In particular, you can now specify a combination of DATABRICKS_CONFIG_PROFILE=my-profile and MLFLOW_REGISTRY_URI=databricks-uc://my-profile before calling agents.deploy() to specify the Databricks CLI profile to use to deploy and access agents.
  • In mlflow.evaluate(), only run retrieval and guidelines metrics if retrieval and guideline context are present, respectively.
  • Add secret-based authentication to clients for mlflow.evaluate().

External groups are now labeled and immutable

December 18, 2024

External groups are groups that are created in Azure Databricks from Microsoft Entra ID. These groups are created using a SCIM provisioning connector and stay in sync with Microsoft Entra ID. External groups are now explicitly labeled as External and can no longer be updated from the Azure Databricks account console or workspace admin settings page by default. To update external group membership from the Azure Databricks UI, an account admin can disable Immutable external groups in the account console preview page.

See Types of groups in Azure Databricks.

vector_search now supports vector embedding inputs

December 17, 2024

The AI Function vector_search now supports vector embedding inputs. You can now use the query_text or query_vector parameters to search for specific text or vector embeddings in vector indexes. See vector_search function.

Specify response formats for ai_query

December 17, 2024

ai_query now supports the responseFormat field for structured outputs. Use responseFormat in your ai_query requests to specify the response format you want the model you are querying to follow. See ai_query function.

Test clean rooms with collaborators within the same metastore

December 17, 2024

You can now test your clean room before full deployment by adding a collaborator from within the same metastore. See Create clean rooms.

Assign compute resources to groups (Public Preview)

December 17, 2024

The new Dedicated access mode (previously Single user) allows you to assign a dedicated all-purpose compute to a group or single user. See Assign compute resources to a group.

This Public Preview also gives your workspace access to the new simplified compute UI. See Use the simple form to manage compute.

A workspace admin must enable this preview. See Manage Azure Databricks Previews.

Delegate the ability to create a storage credential in Unity Catalog to a service principal

December 17, 2024

You can now grant service principals permissions to create a storage credential in a Unity Catalog metastore using the CREATE STORAGE CREDENTIAL privilege. See CREATE STORAGE CREDENTIAL.

Version 2.2 of the Jobs API is released

December 16, 2024

The Jobs API version is updated from 2.1 to 2.2. Updates in version 2.2 of the Jobs API include default queueing of new or updated jobs, and enhancing paging of job and job run responses that include fields with large numbers of values. To learn more about the updates in this version, see Updating from Jobs API 2.1 to 2.2. To see the full Jobs API 2.2 documentation, see Jobs (latest). Although Databricks recommends using version 2.2 of the Jobs API, you can still access versions 2.1 and 2.0. See Jobs (2.1) and Jobs API 2.0.

Unity Catalog MANAGE privilege (Public Preview)

December 14, 2024

You can now grant users the MANAGE privilege on Unity Catalog securable objects. The MANAGE privilege allows users to perform key actions on a Unity Catalog object, including:

  • Managing privileges
  • Dropping the object
  • Renaming the object
  • Transferring ownership

See MANAGE.

Meta Llama 3.3 is now available for provisioned throughput workloads

December 13, 2024

Meta Llama 3.3, a model architecture built and trained by Meta, is now available for serving on Foundation Model APIs provisioned throughput.

View streaming workload metrics for your job runs (Public Preview)

** December 12, 2024**

When you view job runs in the Databricks Jobs UI, you can now view metrics such as backlog seconds, backlog bytes, backlog records, and backlog files for sources supported by Spark Structured Streaming, including Apache Kafka, Amazon Kinesis, and Auto Loader. See View metrics for streaming tasks.

View streaming workload metrics for your Delta Live Tables pipeline updates (Public Preview)

** December 12, 2024**

When you view pipeline updates in the Delta Live Tables UI, you can now view metrics such as backlog seconds, backlog bytes, backlog records, and backlog files for each streaming flow in the pipeline. Streaming metrics are supported for Spark Structured Streaming sources, including Apache Kafka, Amazon Kinesis, and Auto Loader. See View streaming metrics.

Lakehouse Federation supports Oracle (Public Preview)

December 12, 2024

You can now run federated queries on data managed by Oracle. See Run federated queries on Oracle.

Databricks Runtime 16.1 (Beta)

December 11, 2024

Databricks Runtime 16.1 and Databricks Runtime 16.1 ML are now available as Beta releases.

See Databricks Runtime 16.1 and Databricks Runtime 16.1 for Machine Learning

Manage serverless outbound network connections with serverless egress control

December 11th, 2024

Serverless egress control lets you restrict outbound access to specified internet destinations. See What is serverless egress control?.

Network access events system table is now available (Public Preview)

December 11, 2024

Azure Databricks system tables now include a network access table. This table logs an event whenever internet access is denied from your account. To access the table, admins must have the access system schema enabled. See Network access events system table reference.

Monitor and revoke personal access tokens in your account (Private Preview)

December 11, 2024

Account admins can now view a token report to monitor and revoke personal access tokens (PATs) in the account console. Databricks recommends you use OAuth access tokens instead of PATs for greater security and convenience. To join this preview, contact your Azure Databricks account team. See Monitor and revoke personal access tokens in the account.

Unity Catalog can federate to Hive metastores

December 11, 2024

You can now use Unity Catalog to access and govern data that is registered in a Hive metastore. This includes both externally-managed Hive metastores and legacy internal Databricks Hive Hive metastores.

See Hive metastore federation: enable Unity Catalog to govern tables registered in a Hive metastore.

Remove metastore-level storage to enforce catalog-level storage isolation

December 11, 2024

If you have metastore-level storage for managed tables and volumes (also known as the metastore storage root), but you want to enforce data storage isolation at the catalog or schema level, you can now remove that metastore-level storage without interrupting existing workloads. See Remove metastore-level storage.

Meta Llama 3.3 70B Instruct is now available on Model Serving

December 11, 2024

Mosaic AI Model Serving now supports Meta Llama 3.3 70B Instruct, a state-of-the-art large language model built and trained by Meta. Llama 3.3 70B Instruct is available as part of Foundation Model APIs pay-per-token. This availability also includes support on Function Calling.

Starting December 11, 2024, Meta-Llama-3.1-70B-Instruct replaces support for Meta-Llama-3-70B-Instruct in Foundation Model APIs pay-per-token endpoints.

bamboolib is now deprecated

December 10, 2024

bamboolib is now deprecated. Users can still access bamboolib to perform low-code data analysis within notebooks, but Databricks is no longer actively developing nor supporting this tool. For assistance with code generation, use the Databricks Assistant.

Streamline AI Agent Evaluation using synthetic evaluation sets

December 9, 2024

Evaluate your AI agent by generating a representative evaluation set from your documents. The synthetic generation API is tightly integrated with Agent Evaluation, allowing you to quickly evaluate and improve the quality of your agent’s responses without going through the costly process of human labeling. See Synthesize evaluation sets.

Improve Databricks-to-Databricks Delta Sharing table read performance with history sharing (Public Preview)

December 5, 2024

Improve performance for Databricks-to-Databricks table shares by enabling history sharing. See Improve table read performance with history sharing.

Maximum personal access token lifetime now 730 days (two years)

December 5, 2024

The default maximum lifetime for newly created Databricks-issued personal access tokens is now set to 730 days (two years). Previously, personal access tokens could be created with no expiration by default. With this update, users cannot generate new tokens with a lifetime exceeding 730 days, and tokens created without a specified lifetime are set to a 730-day duration. If you configured the maximum token lifetime for your workspace to less than 730 days, the configuration remains unchanged. See Monitor and revoke personal access tokens and Azure Databricks personal access token authentication.

Mosaic AI Model Training - serverless forecasting (Public Preview)

December 5, 2024

Mosaic AI Model Training - forecasting improves upon the existing AutoML forecasting experience with managed serverless compute, Unity Catalog support, access to deep learning algorithms, and an upgraded interface. See Forecasting (serverless) with AutoML.

Add budget policies to model serving endpoints

December 4, 2024

Budget policies are now supported on model serving endpoints. See Manage model serving endpoints.