How to find empty folder and empty files by using synapse

Zhu, Yueli YZ [NC] 255

Hi,

How can I find if there are empty folder or files in gen2 container by using synapse? Either sql or notebook python. Thanks!

Ganesh Gurram 1,510 Reputation points Microsoft Vendor

2024-11-19T20:44:02.1833333+00:00
@Zhu, Yueli YZ [NC] - Thanks for the question and using MS Q&A forum.

To find empty folders and empty files in an Azure Data Lake Storage Gen2 container using Azure Synapse Analytics, you can use Python notebook. Here's how you can achieve this:

In a Synapse notebook, you can use the Azure Storage Blob library to list the files and folders and check their sizes. Here's an example of how you can do it:

Install the required library (if not already installed):

!pip install azure-storage-blob

Use the following Python code to list and check empty files and folders:

from azure.storage.blob import BlobServiceClient, ContainerClient import os # Replace with your connection string connection_string = "your_connection_string" container_name = "your_container_name" # Initialize the BlobServiceClient blob_service_client = BlobServiceClient.from_connection_string(connection_string) container_client = blob_service_client.get_container_client(container_name) # List blobs in the container blob_list = container_client.list_blobs() empty_files = [] empty_folders = set() for blob in blob_list: blob_client = container_client.get_blob_client(blob) props = blob_client.get_blob_properties() # Check if it is a file and is empty if props.size == 0: empty_files.append(blob.name) # Check if it is a folder (folders in ADLS Gen2 are virtual and based on naming conventions) # Here we assume folders end with '/' if blob.name.endswith('/') and props.size == 0: empty_folders.add(blob.name) print("Empty Files:") for file in empty_files: print(file) print("\nEmpty Folders:") for folder in empty_folders: print(folder)

This script will print the list of empty files and folders. Note that folders in Azure Data Lake Storage Gen2 are virtual and usually based on naming conventions, so this script considers folders as those blobs that end with a '/'.

Make sure to replace "your_connection_string" and "your_container_name" with your actual connection string and container name. This approach uses Azure Storage Blob library to interact with Azure Data Lake Storage Gen2 and can be run in an Azure Synapse notebook.

Hope this helps. Do let us know if you have any further queries!
Ganesh Gurram 1,510 Reputation points Microsoft Vendor

2024-11-20T18:23:49.06+00:00

@Zhu, Yueli YZ [NC] - We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Ganesh Gurram 1,510 Reputation points Microsoft Vendor

2024-11-21T13:09:31.3633333+00:00

@Zhu, Yueli YZ [NC] - Following up to see if the above suggestion was helpful. And, if you have any further query do let us know.

Share via

How to find empty folder and empty files by using synapse

Your answer