We want to loop all folder and file recursively of BLOB Storage for given container using python in azure data factory custom activity and write data in in blob in .CSV file

Question

Hi All,

We want to loop all folder and file recursively of BLOB Storage for given container using python in azure data factory custom activity and write data in in blob in .CSV file.

also when we connect blob storage using ADF custom activity using SAS Key instead of blob access key

Answer 1

Hi manish verma ,

Thankyou for using Microsoft Q&A platform and thanks for posting your query here.

To use SAS key for connection with blob, kindly generate a SAS Token for your Blob Storage container with list and write permissions. Include the SAS token in the connection string for authentication.

Here is the sample code you can try to iterate through the folders in blob :

from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient
import csv
import io

# Parameters
sas_token = "<YOUR_SAS_TOKEN>"  # Replace with your SAS token
storage_account_url = "https://<YOUR_STORAGE_ACCOUNT_NAME>.blob.core.windows.net/"
container_name = "<YOUR_CONTAINER_NAME>"
output_blob_name = "output.csv"

# Initialize BlobServiceClient
blob_service_client = BlobServiceClient(account_url=storage_account_url, credential=sas_token)
container_client = blob_service_client.get_container_client(container_name)

# Function to list all blobs in the container recursively
def list_blobs_recursively(container_client):
    blob_data = []
    for blob in container_client.list_blobs():
        blob_data.append({"name": blob.name, "size": blob.size, "last_modified": blob.last_modified})
    return blob_data

# Fetch all blobs
blobs = list_blobs_recursively(container_client)

# Write blob details to a CSV
output = io.StringIO()
csv_writer = csv.DictWriter(output, fieldnames=["name", "size", "last_modified"])
csv_writer.writeheader()
csv_writer.writerows(blobs)

# Upload CSV to Blob Storage
blob_client = container_client.get_blob_client(output_blob_name)
blob_client.upload_blob(output.getvalue(), overwrite=True)

print(f"CSV file '{output_blob_name}' created successfully and uploaded to Blob Storage.")

Hope it helps. Thankyou