November 2023
These features and Azure Databricks platform improvements were released in November 2023.
Note
Releases are staged. Your Azure Databricks workspace might not be updated until a week or more after the initial release date.
Mosaic AI Vector Search is Public Preview
November 30, 2023
Mosaic AI Vector Search is a serverless similarity search engine that allows you to store a vector representation of your data, including metadata, in a vector database. As part of this functionality, you can create auto-updating vector search indexes from data in Unity Catalog and query them with a simple API to return the most similar vectors as part of your RAG workflows. See Mosaic AI Vector Search.
Access controls lists can no longer be disabled
November 29, 2023
The following settings are enabled on all Premium plan workspaces and can no longer be disabled from the workspace admin settings page or the Workspace Conf API:
- Workspace access control: enables users to configure permissions to workspace objects.
- Cluster, pool, jobs access control: enables users to configure permissions to clusters, pools, and jobs.
- Cluster visibility control: prevents clusters from being visible to users that have no permissions on them.
- Jobs visibility control: prevents jobs being from visible to users that have no permissions on them.
- Workspace visibility control setting: prevents notebooks and folders from being from visible to users that have no permissions on them.
Access control settings have been enabled on new workspaces by default since March 2018 and visibility control settings have been enabled by default since December 2020.
For more information, see Access control lists.
New behaviors and actions in Catalog Explorer for volumes
November 22, 2023
You can now create a new volume in the target schema if none exists from the Upload to volume UI. See Upload files to a Unity Catalog volume.
Actions such as Delete file and Download file have been added to Catalog Explorer for files in volumes.
Databricks Runtime 14.2 is GA
November 22, 2023
Databricks Runtime 14.2 and Databricks Runtime 14.2 ML are now generally available.
See Databricks Runtime 14.2 (EoS) and Databricks Runtime 14.2 for Machine Learning (EoS).
Databricks SQL Connector for Python version 3.0.0
November 17, 2023
You can now run pip install databricks-sql-connector==3.0.0
.
This release:
- Removes support for Python 3.7
- Adds support for native parameterized SQL queries. Requires DBR 14.2 and above. For more information, see Using Native Parameters in the databricks-sql-python GitHub repo.
- Includes our completely rewritten SQLAlchemy dialect:
- Adds support for SQLAlchemy >= 2.0 and drops support for SQLAlchemy 1.x
- Full end-to-end test coverage of all supported features
- Detailed usage notes available here
- Adds support for:
- New types:
TIME
,TIMESTAMP
,TIMESTAMP_NTZ
,TINYINT
Numeric
type scale and precision, likeNumeric(10,2)
- Reading and writing
PrimaryKeyConstraint
andForeignKeyConstraint
- Reading and writing composite keys
- Reading and writing from views
- Writing
Identity
to tables (autoincrementing primary keys) LIMIT
andOFFSET
for paging through results- Caching metadata calls
- New types:
- Enables Cloud Fetch by default. To disable, set
use_cloud_fetch=False
when buildingdatabricks.sql.client
. - Add integration tests for Azure Databricks Unity Catalog volumes ingestion queries
- Retries:
- Add
_retry_max_redirects
config - Set
_enable_v3_retries=True
and warn if users override it.
- Add
- Security: Bumps minimum pyarrow version to 14.0.1 (CVE-2023-47248)
Libraries in workspace files supported on no-isolation shared clusters
November 15, 2023
You can now install libraries stored as workspace files to compute configured with no-isolation shared access mode in Databricks Runtime 14.1 and above.
Deprecation of workspace libraries
November 15, 2023
Workspace libraries are now deprecated and creating new workspace libraries has been disabled. Existing workspace libraries continue to function. Note that storing libraries as workspace files is distinct from workspace libraries and is still fully supported. You can install libraries stored as workspace files directly to compute or job tasks. See Libraries.
Support for Azure confidential computing VM types is now GA
November 15, 2023
Azure Databricks support for Azure confidential computing VM types is now generally available. Azure confidential computing helps protect data in use, preventing the cloud provider from having access to sensitive data. See Azure confidential computing VMs.
Search for Databricks Marketplace listings using global workspace search
November 13, 2023
You can now find data sets, AI models, and Databricks Solution Accelerators in Databricks Marketplace using the global search bar in the Azure Databricks workspace UI. See What is Databricks Marketplace? and Search for workspace objects.
Consume data products in Databricks Marketplace using external platforms
November 10, 2023
Consumers without a Unity Catalog-enabled Azure Databricks workspace can now access data products in Databricks Marketplace. You can use Delta Sharing open sharing connectors to access Marketplace data using a number of common platforms, including Microsoft Power BI, Microsoft Excel, pandas, Apache Spark, and non-Unity Catalog Databricks workspaces. Only tabular data sets are available on external platforms (not Databricks notebooks or models, for example). See Access data products in Databricks Marketplace using external platforms.
Support for serverless SQL warehouses added for several Azure regions
November 9, 2023
The following Azure regions now support serverless SQL warehouses:
- Brazil South (brazilsouth)
- Central India (centralindia)
- Japan East (japaneast)
- North Central US (northcentralus)
- UK South (uksouth)
See Azure Databricks regions and Enable serverless SQL warehouses.
Automatic enablement of Unity Catalog for new workspaces
November 9, 2023
Databricks has begun to enable Unity Catalog automatically for new workspaces. This removes the need for account admins to configure Unity Catalog after a workspace is created. Rollout will proceed gradually across accounts. See Automatic enablement of Unity Catalog.
Databricks Runtime 14.2 (beta)
November 8, 2023
Databricks Runtime 14.2 and Databricks Runtime 14.2 ML are now available as beta releases.
See Databricks Runtime 14.2 (EoS) and Databricks Runtime 14.2 for Machine Learning (EoS).
Databricks Marketplace includes Databricks Solution Accelerators
November 7, 2023
Databricks Marketplace now gives you access to Databricks Solution Accelerators, purpose-built guides that provide fully functional notebooks, best practices guidance, and sample data to get you set up quickly with many popular use cases on Azure Databricks. See Get access to Databricks Solution Accelerators.
Lakehouse Federation adds support for Google BigQuery
November 3, 2023
You can now configure Lakehouse Federation to run federated queries with the BigQuery
connector type. The new BigQuery connector is supported on Pro and Serverless SQL warehouses and Databricks Runtime 13.3 LTS and above. See Run federated queries on Google BigQuery.