Drop Delta table features

Important

Support for dropping Delta table features and downgrading protocol versions is in Public Preview in Databricks Runtime 14.1 and above.

Azure Databricks provides limited support for dropping table features. To drop a table feature, the following must occur:

• Disable table properties that use the table feature.
• Remove all traces of the table feature from the data files backing the table.
• Remove transaction entries that use the table feature from the transaction log.
• Downgrade the table protocol.

Where supported, you should only use this functionality to support compatibility with earlier Databricks Runtime versions, Delta Sharing, or other Delta Lake reader or writer clients.

Important

All DROP FEATURE operations conflict with all concurrent writes.

Streaming reads fail when they encounter a commit that changes table metadata. If you want the stream to continue you must restart it. For recommended methods, see Production considerations for Structured Streaming.

How can I drop a Delta table feature?

To remove a Delta table feature, you run an ALTER TABLE <table-name> DROP FEATURE <feature-name> [TRUNCATE HISTORY] command. See ALTER TABLE.

You must use Databricks Runtime 14.1 or above and have MODIFY privileges on the target Delta table.

What Delta table features can be dropped?

You can drop the following Delta table features:

• checkConstraints. See Constraints on Azure Databricks.
• collations-preview. See Collation support for Delta Lake.
• columnMapping. See Rename and drop columns with Delta Lake column mapping.
• deletionVectors. See What are deletion vectors?.
• typeWidening-preview. See Type widening.
• v2Checkpoint. See Compatibility for tables with liquid clustering.

You cannot drop other Delta table features.

Enable table features to drop legacy features

The DROP FEATURE command requires protocol versions that support table feature reads and writes. Delta features like columnMapping and checkConstraints were supported in earlier protocol versions. Depending on other features enabled on the table, you might need to upgrade protocol versions before you can drop these features.

You can use the following command to upgrade the table reader and writer versions, which allows you to drop column mapping and downgrade the protocol:

ALTER TABLE <table-name> SET TBLPROPERTIES (
  'delta.minReaderVersion' = '3',
  'delta.minWriterVersion' = '7'
)

How are Delta table features dropped?

Because Delta table features represent reader and writer protocols, they must be completely absent from the transaction log for full removal. Dropping a feature occurs in two stages and requires time to elapse before completion. The specifics of feature removal vary by feature, but the following section provides a general overview.

Prepare to drop a table feature

During the first stage, the user prepares to drop the table feature. The following describes what happens during this stage:

1. The user runs the DROP FEATURE command.
2. Table properties that specifically enable a table feature have values set to disable the feature.
3. Table properties that control behaviors associated with the dropped feature have options set to default values before the feature was introduced.
4. As necessary, data and metadata files are rewritten respecting the updated table properties.
5. The command finishes running and returns an error message informing the user they must wait 24 hours to proceed with feature removal.

After first disabling a feature, you can continue writing to the target table before completing the protocol downgrade, but cannot use the table feature you are removing.

Note

If you leave the table in this state, operations against the table do not use the table feature, but the protocol still supports the table feature. Until you complete the final downgrade step, the table is not readable by Delta clients that do not understand the table feature.

Downgrade the protocol and drop a table feature

To drop the table feature, you must remove all transaction history associated with the feature and downgrade the protocol.

1. After at least 24 hours have passed, the user executes the DROP FEATURE command again with the TRUNCATE HISTORY clause.
2. The client confirms that no transactions in the specified retention threshold use the table feature, then truncates the table history to that treshold.
3. The protocol is downgraded, dropping the table feature.
4. If the table features that are present in the table can be represented by a legacy protocol version, the minReaderVersion and minWriterVersion for the table are downgraded to the lowest version that supports exactly all remaining features in use by the Delta table.

Important

Running ALTER TABLE <table-name> DROP FEATURE <feature-name> TRUNCATE HISTORY removes all transaction log data older than 24 hours. After dropping a Delta table feature, you do not have access to table history or time travel.

See How does Azure Databricks manage Delta Lake feature compatibility?.