January 2024
These features and Azure Databricks platform improvements were released in January 2024.
Note
Releases are staged. Your Azure Databricks account might not be updated until a week or more after the initial release date.
Native XML file format support (Public Preview)
January 31, 2024
Native XML file format support is now in Public Preview. XML file format support enables ingestion, querying, and parsing of XML data for batch processing or streaming. It can automatically infer and evolve schema and data types, supports SQL expressions like from_xml, and can generate XML documents. It doesn’t require external jars and works seamlessly with Auto Loader, read_files, COPY INTO, and Delta Live Tables. For more information, see Read and write XML files.
Share AI models using Databricks Marketplace (Public Preview)
January 31, 2024
You can now use Marketplace to share models that are registered in Unity Catalog. Both the provider and consumer workspaces must be enabled for Unity Catalog to participate in model sharing.
See What is Databricks Marketplace? and Add models to a share.
Updates to network security group rules
January 31, 2023
To improve platform scalability, the Azure Databricks control plane now requires new endpoints for the classic data plane to connect to. For workspaces deployed in your own Azure virtual network, the network ports 3306 and 8443-8451 are now added to the existing security rule to the AzureDatabricks destination, enforcing outbound traffic on port 443.
For more information, see Network security group rules.
Workspace path update
Historically, users were required to include the /Workspace path prefix for some Databricks APIs (%sh) but not for others (%run, REST API inputs). Now, users can provide workspace paths with the /Workspace prefix everywhere in Databricks. Old references to paths without the /Workspace prefix are redirected and will continue to work. Databricks recommends that all workspace paths use the /Workspace prefix to differentiate them from Volume and DBFS paths.
Support for Azure Storage firewall from serverless compute is GA
January 30, 2024
Support for Azure Storage firewall from Azure Databricks serverless compute is GA. You can enforce that serverless SQL warehouses use stable subnet IDs to connect to Azure Storage and allowlist those identifiers on your Azure resource firewall. See Configure a firewall for serverless compute access.
Streamlined creation of Azure Databricks jobs
January 29, 2024
An empty job is now created immediately after you click the button to create a job, allowing you to edit job-level settings such as tags, schedule, or notification settings before adding the first task. Previously, you had to create the first task before getting access to job-level settings. See Configure and edit Databricks Jobs.
Monitor GPU model serving workloads using inference tables
January 29, 2024
You can now monitor your GPU model serving workloads using inference tables. See Inference tables for monitoring and debugging models.
URI path-based access to Unity Catalog external volumes
January 25, 2024
You can now use cloud storage URIs for path-based access to data governed by Unity Catalog stored in external volumes. See What path is used for accessing files in a volume?.
Access controls lists can be enabled on upgraded workspaces
January 24, 2024
Access control settings were enabled on all Premium plan workspaces in November 2023.
For workspaces that upgrade from the Standard plan to the Premium plan, access control settings are disabled by default. Workspace admins can now enable the following access control settings on upgraded workspaces:
- Workspace access control: Enables users to configure permissions to workspace objects.
- Cluster, pool, jobs access control: Enables users to configure permissions to clusters, pools, and jobs.
- Cluster visibility control: Prevents clusters from being visible to users who have no permissions on them.
- Jobs visibility control: Prevents jobs from being visible to users who have no permissions on them.
- Workspace visibility control setting: Prevents notebooks and folders from being visible to users who have no permissions on them.
After an access control setting is enabled, it cannot be disabled. For more information, see Access control lists.
Marketplace listing events system table now available (Public Preview)
January 24, 2024>
The Marketplace system schema now includes a listing funnel events table (system.marketplace.listing_funnel_events) where you can monitor consumer actions on your listings. See Listing funnel events table.
Updated UI for notebook cells (Public Preview)
January 24, 2024
An updated notebook cell design is available. To enable the new cell design, open the editor settings page in the workspace. In the sidebar, click Developer. Under Experimental features, toggle New cell UI. For details about working with notebook cells see Databricks notebook interface and controls.
Quick Fix help with syntax errors in the notebook
January 24, 2024
You can get help from Databricks Assistant for syntax errors by using Quick Fix. For details, see Python error highlighting.
Databricks Runtime 14.3 LTS (Beta)
January 22, 2024
Databricks Runtime 14.3 LTS and Databricks Runtime 14.3 LTS ML are now available as Beta releases.
See Databricks Runtime 14.3 LTS and Databricks Runtime 14.3 LTS for Machine Learning.
Share AI models using Delta Sharing (Public Preview)
January 19, 2024
You can now use Delta Sharing to share models that are registered in Unity Catalog. Both the provider and recipient workspaces must be enabled for Unity Catalog to share models.
Databricks Marketplace supports volume sharing
January 18, 2023
Databricks Marketplace now gives you access to volumes (non-tabular data) as long as you have a Unity Catalog-enabled Databricks workspace. Volumes are Unity Catalog objects that represent a logical volume of storage in a cloud object storage location, and they provide a way to manage governance and sharing of non-tabular data assets.
See Access data products in Databricks Marketplace (Unity Catalog-enabled workspaces).
Create widgets from the Databricks UI
January 18, 2024
You can now create Databricks Widgets using the Databricks UI. For details, see Create widgets using the UI.
Warehouse events system table is now available (Public Preview)
January 18, 2024
The system.compute.warehouse_events table records when SQL warehouses start, stop, scale up and down. You can use the table to monitor and manage the SQL warehouses in your workspaces. See Warehouse events system table reference.
Workspace file size limit is now 500MB
January 15, 2023
Individual workspace files are now limited to 500 MB. Previously, the limit was 200 MB. For more details, read Limitations.
Feature removal notice for legacy Git integration in Databricks
January 12, 2024
After January 31st, 2024, Databricks will remove legacy notebook Git integrations. This feature has been in legacy status for more than two years, and a deprecation notice has been displayed in the product UI since November 2023.
For details on migrating to Databricks Git folders (Repos), read Switching to Databricks Repos from Legacy Git integration. If this removal impacts you and you need an extension, contact your Databricks Account Executive.
Databricks ODBC driver 2.7.7
January 9, 2024
We have released version 2.7.7 of the Databricks ODBC driver (download). This release adds the following new features and enhancements:
- Azure Managed Identity authentication.
- Support for asynchronous metadata operations. The server needs to support
SPARK_CLI_SERVICE_PROTOCOL_V9and above. - Support for Parameterized Query in the Native Query Mode. The server needs to support
SPARK_CLI_SERVICE_PROTOCOL_V8and above. - OAuth enhancement: The connector can now disable the token cache for the browser-based authentication flow.
This release also resolves the following issues in 2.7.7:
- The connector ignores private root CA when using the Windows Trust Store.
- When an operation fails, the connector does not close the heartbeat threads.
- The connector does not support the optional refresh token.
- When querying tables with foreign names, the connector fails to read them.
- When using OAuth authentication, if the string length of the HOST is less than 20, the connector terminates unexpectedly.
Databricks Runtime 13.2 series support ends
January 6, 2024
Support for Databricks Runtime 13.2 and Databricks Runtime 13.2 for Machine Learning ended on January 6. See Databricks support lifecycles.
AI assistive features are enabled by default
January 2-8, 2024
AI assistive features are now enabled by default for workspaces without the compliance security profile enabled. These include:
Admins can opt out of AI assistive features for all workspaces in their account or a subset of workspaces. See For an account: Disable or enable Databricks Assistant features.