How do I configure my Azure ML Workspace such that when I delete experiments and jobs, the corresponding Blobs in Azure Blob Storage are also deleted

Chiran Bopitiya 0

I regularly submit jobs and experiments to train ML models. With each experiment, all my files and outputs are uploaded to my blob storage account. I have been constantly deleting failed and cancelled experiments, but I found out that deleting experiments via the Azure ML workspace frontend does not actually delete the files in the Blob Storage and I am still paying for the storage costs of deleted experiments. I have so many experiments, many of which I wish to delete and some important ones that I wish to keep. It is extremely tedious to manually delete all of them one by one via the Azure Storage Explorer. How do I configure my Azure ML workspace such that when I delete an experiment, the files from Blob Storage are also deleted?

kothapally Snigdha 1,635 Reputation points Microsoft External Staff

2025-03-05T22:39:35.06+00:00

Hi Chiran Bopitiya

Just checking to see if the below response was helpful.
kothapally Snigdha 1,635 Reputation points Microsoft External Staff

2025-03-07T00:11:01.65+00:00

Hi Chiran Bopitiya

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Chiran Bopitiya 0 Reputation points

2025-03-07T04:54:31.9733333+00:00
Hi @kothapally Snigdha

It was not very helpful. In fact, im pretty sure my question was just copied and pasted into some AI to generate a sample script. For instance, this line of code:

for exp in experiments: if exp.archived:

exp is a string and would not have the .archive attribute.

The response does not take into account how experiments are saved in Blob storages. Also, I was hoping there was an option to configure in Azure ML settings where deleted experiments are also deleted from blob storage, but it seems like this option does not exist.

I will write my own script to resolve this. thanks.
SriLakshmi C 3,015 Reputation points Microsoft External Staff

2025-03-07T07:37:31.03+00:00
Hi Chiran Bopitiya,

In Azure Machine Learning (Azure ML) Workspace, deleting experiments or jobs through the UI or SDK does not automatically remove associated blob storage files. By default, Azure ML does not clean up blob storage to prevent accidental data loss. However, you can configure and automate the deletion process to manage storage efficiently.

When you run experiments in Azure ML, data, logs, and model artifacts are stored in Azure Blob Storage within the workspace’s default storage account. Deleting an experiment only removes metadata but does not delete the actual files, leaving them in storage.

To automate this cleanup, Azure Storage Lifecycle Management allows you to define rules for deleting old blobs. You can set up a Lifecycle Policy in the Azure Portal by navigating to Storage Account → Blob Storage → Lifecycle Management under the Data Management section. Then, click on “Add a rule” and configure it as follows:

Name: "Delete ML experiment blobs"

Rule Scope: Limit to the azureml container (or the relevant storage location).

Conditions:

Delete blobs older than X days.

Target blobs with specific prefixes (e.g., /experiment-runs/).

Finally, save the policy, and Azure will automatically remove old experiment files based on the configured rules. This approach helps optimize storage usage and reduces unnecessary costs while maintaining a structured ML workspace.

Could you please take a moment to retake the survey on the above response? Your feedback is greatly appreciated.

Thank you!
Chiran Bopitiya 0 Reputation points

2025-03-07T08:43:43.39+00:00
Hi @SriLakshmi C ,

Im aware of the workflow with regards to Lifecycle Management.

This only solves a part of the problem.

When a new job is submitted, a few things happen

A new container is created to store the scripts that the user has uploaded for the job

Output files and logs are stored in the azureml container

Screenshot 2025-03-07 at 4.37.00 PM.png

If you see the screenshot of my storage explorer, on the right side you will see all the job outputs and logs that are stored in the azureml container. If I use the Lifecycle Management workflow, those will be removed.

However, if you see on the left, all the containers that start with cd7c are all the scripts that were uploaded for each job. It seems that for every job that I submit, a new container is created to store my files

Screenshot 2025-03-07 at 4.40.53 PM.png

Is the Lifecycle Management workflow able to remove all these containers of deleted experiments? The scripts used to run the experiment and the outputs/logs are stored in different locations.
Chiran Bopitiya 0 Reputation points

2025-03-07T08:56:17.6833333+00:00

Also I just realized @SriLakshmi C , the scope of the rule that you proposed doesn't actually resolve the problem, as it will simply delete all blobs older than X days or with specific prefixes. I don't wish to delete all blobs older than X days, as there are some experiments I wish to keep. As for the prefix matching, does that mean I will have to change the experiment name to a match the prefix as specified in the rule for it to be deleted? That doesn't seem like a very feasible workflow either. Is there a rule where blob storage deletes the files of experiments that have been deleted via the front-end UI of Azure ML?
VasaviLankipalle-MSFT 18,566 Reputation points

2025-03-08T01:05:19.4066667+00:00

Hello @Chiran Bopitiya , I don't think there's a rule that deletes blob files if experiments have been deleted via Azure ML.

I would suggest you explore this documentation to see if the filter set and filter set rule might help: https://learn.microsoft.com/en-us/azure/storage/blobs/lifecycle-management-overview?tabs=azure-portal

1 answer

Amira Bedhiafi 29,711

You can automate the deletion of blobs associated with deleted experiments using a script.

You can schedule this script to run periodically using Azure Functions, Azure Automation, or even a simple cron job on a virtual machine.

from azureml.core import Workspace
from azure.storage.blob import BlobServiceClient, ContainerClient
import os

# Load your Azure ML workspace
ws = Workspace.from_config()

# Get the default datastore (usually the blob storage account)
default_ds = ws.get_default_datastore()

# Connect to the blob storage account
blob_service_client = BlobServiceClient(account_url=default_ds.account_url, credential=default_ds.credential)
container_client = blob_service_client.get_container_client(default_ds.container_name)

# List all experiments
experiments = ws.experiments

# Iterate through experiments and delete blobs for deleted experiments
for exp in experiments:
    if exp.archived:  # Check if the experiment is deleted/archived
        print(f"Deleting blobs for experiment: {exp.name}")
        # List blobs in the experiment's folder
        blobs = container_client.list_blobs(name_starts_with=f"experiments/{exp.name}/")
        for blob in blobs:
            print(f"Deleting blob: {blob.name}")
            container_client.delete_blob(blob.name)

print("Blob cleanup completed.")

Share via

How do I configure my Azure ML Workspace such that when I delete experiments and jobs, the corresponding Blobs in Azure Blob Storage are also deleted

1 answer

Your answer