Investigating Missing Files in Azure Blob Storage

Question

Recently, we noticed that some files are missing from our Azure Blob Storage container, even though no manual deletions were performed. Our goal is to find the root cause of the missing files, identify the exact area where they went missing, and determine which action (e.g., Databricks, automated scripts, or frontend application code) caused the deletion.

We have already analyzed Azure Monitor Logs, Databricks Audit Logs, and Job History, but unfortunately, we could not find any clue as to what caused the deletions. We are also checking Soft Delete and Versioning settings to explore potential recovery options.

If anyone has encountered similar issues or has insights on how to effectively track and pinpoint file deletions in Azure Blob Storage, please guide me on how to solve this issue. Your suggestions will be greatly appreciated!

Answer 1

Hi Rajasekar Vudhi,

To investigate missing files in Azure Blob Storage, you can utilize the server-side logs to identify operations that may have led to the deletions. The server-side Storage Logging log can show when a client deleted an object, including the operation type and requested object key. This can help you determine if another user or process was responsible for the deletion.

Additionally, examining the client-side logs generated by the Storage Client Library can provide insights into specific requests made to the storage service. These logs can help you trace back the actions leading to the missing files.

 

Storage Account Activity Log: This log captures control plane operations, Check the Storage Account Activity Log for control plane operations, which can identify changes made to the storage account.

https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/activity-log?tabs=powershell

Data Plane Operations: These operations are captured in Azure resource logs for Storage. Configuring Diagnostic settings to export logs to a Log Analytics workspace can facilitate a more detailed query experience.

https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/diagnostic-settings?tabs=portal

Soft Delete: Ensure soft delete is enabled to protect blobs from accidental deletions.

Blob Versioning: Verify if blob versioning is enabled to maintain previous versions of blobs.

Setup alerts to notify you of any delete operations or unusual activities.

https://learn.microsoft.com/en-us/troubleshoot/azure/azure-storage/blobs/alerts/storage-monitoring-diagnosing-troubleshooting?tabs=dotnet#client-previously-deleted-the-object

Hope the above suggestion helps! Please let us know do you have any further queries.

Please do consider to “Accept the answer” wherever the information provided helps you, this can be beneficial to other community members. 

Answer 2

Hi Rajasekar Vudhi,

Thanks for reaching out.

Yes, for the missing data, if it has been deleted, the best way is to analyze using Storage Diagnostic logging that shall help you tracking based on the available fields such as Client IP, User Agent, Auth method used etc,

Below are the 2 blobs that will be helpful in the use case.

https://techcommunity.microsoft.com/blog/azurepaasblog/who-deleted-a-blob/3941404

https://techcommunity.microsoft.com/blog/azurepaasblog/leveraging-storage-analytics-logs-to-analyze-who-accessed-the-storage-account/2173539

Hope this helps! Please let us know do you have any further queries.

Please do consider to “up-vote” wherever the information provided helps you, this can be beneficial to other community members.