Data size of databricks delta tables

Question

It has been observed that the size of delta tables are much less as compared to when checked the underlying delta files in the storage account.

Suppose a databricks delta table raw.deltaTableA has size of 2MB if we check the size of underlying delta files directly in data lake it shows 50MB.

How this data size is being calculated ?

Answer 1

Hi NIKHIL KUMAR,

Thanks for reaching out to Microsoft Q&A.

A delta table is a high-level abstraction that represents your data in a structured format. It includes metadata, schema information, and transaction logs. Delta files, on the other hand, are the actual data files stored in the underlying storage such as Azure Data Lake Storage. These files contain the raw data in a columnar format (Parquet or Delta format) hence the size you are seeing might be higher than the table.

Please 'Upvote'(Thumbs-up) and 'Accept' as an answer if the reply was helpful. This will benefit other community members who face the same issue.

Answer 2

The delta table size on Databricks will always show the size of the latest snapshot or version of that delta table.

The under lying storage has all the data for the previous versions of the tables. This is to allow time travel to a previous version of the table which is a standard feature of Delta tables. When we do a regular vacuum, versions that are older than 7 days are deleted thereby limiting the time travel to 7 days in the past. This reduces the storage occupied and is a recommended approach.