How to Move Files Between Azure File Shares from Transaction Optimized to Cool Tier?

Question

Scenario:

I have two Azure storage accounts configured as follows:

1. Premium Storage Account (Transactional Optimized Tier)
   • File Share Name: main
   1. Standard Storage Account (Cool Tier)
      • File Share Name: archive

Since Azure File Share Lifecycle Management is not supported, I need a workaround to move files that haven't been accessed for X number of days from the transaction optimized file share (main) to the cool-tier file share (archive).

Requirements:

✅ Identify files in the transaction optimized file share that have not been accessed for X days ✅ Move these files to the cool-tier file share in a different storage account ✅ Ensure the files are deleted from the source after a successful transfer ✅ Automate this process using PowerShell

Questions:

1. What is the best approach to achieve this in Azure?
2. Are there any built-in services that can automate this process?
3. Can you provide a PowerShell script to automate this file movement?

Answer 1

Hi Yasar Shaikh,

Using Azure Data Factory for automating the file movement between Azure File Shares. Azure Data Factory provides a robust and scalable way to handle data movement and transformation.

Steps to Set Up Azure Data Factory for File Movement

Create an Azure Data Factory Instance:

• Go to the Azure portal and create a new Data Factory instance.

Create Linked Services:

• Create linked services for both the source (transaction optimized file share) and the destination (cool-tier file share) storage accounts.

Create Datasets:

• Create datasets for the source and destination file shares.

Create a Pipeline:

• Create a new pipeline in Azure Data Factory.
• Add a Copy Data activity to the pipeline.
• Configure the source and destination datasets in the Copy Data activity.

Add a Filter Activity:

• Use a filter activity to filter files based on the last access time.
• You can use a custom activity or a stored procedure to filter files that haven't been accessed for a specified number of days.

Add a Delete Activity:

• After the files are copied to the destination, add a delete activity to remove the files from the source.

Example Pipeline Configuration

Here’s a high-level overview of how the pipeline might look:

Copy Data Activity:

• Source: Transaction optimized file share dataset.
• Destination: Cool-tier file share dataset.

Filter Activity:

• Use a custom activity to filter files based on the last access time.

Delete Activity:

• Delete files from the source file share after successful copy.

Automate the Pipeline

• Schedule the pipeline to run at regular intervals using triggers in Azure Data Factory.

Additional Resources

• Azure Data Factory Documentation

Copy Data Tool in Azure Data Factory

If you have any other questions or are still running into more issues, let me know in the "comments" and I would be happy to help you.