how to order sequence of python scrip using batch pool

Question

I have stored three python scripts in Blob Storage and I have created a Batch account and I am about to create the “pool”. But these python scripts has to be run in particular order. Hence my question: how do I order the sequence of the python script to be ran using the batch pool ?

Answer 1

Hello Renan,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

I understand that you would like to know how you can order sequence of python script using batch pool.

Job dependencies is the best practices to run your Python scripts in a specific order using Azure Batch.

By following the below steps, you will achieve the purpose:

• Ensure you have your Batch pool set up with the necessary compute nodes.
• Make sure your Python scripts are uploaded to Azure Blob Storage.
• Create a job in Azure Batch that will contain the tasks (your Python scripts).

Define Tasks with Dependencies:

Task 1: Create the first task to run the first Python script.

Task 2: Create the second task to run the second Python script and set it to depend on the completion of Task 1.

Task 3: Create the third task to run the third Python script and set it to depend on the completion of Task 2.

The below is an example of the code to sets up a job with three tasks, each dependent on the completion of the previous one. Make sure to replace placeholders like 'your_batch_account_name', 'your_batch_account_key', 'your_batch_account_url', 'your_pool_id', and 'your_blob_storage_url' with your actual Azure details:

from azure.batch import BatchServiceClient
from azure.batch.batch_auth import SharedKeyCredentials
from azure.batch.models import *
# Initialize Batch client
batch_client = BatchServiceClient(
    credentials=SharedKeyCredentials('your_batch_account_name', 'your_batch_account_key'),
    batch_url='https://your_batch_account_url'
)
# Create job
job_id = 'my_job'
batch_client.job.add(JobAddParameter(id=job_id, pool_info=PoolInformation(pool_id='your_pool_id')))
# Define tasks
tasks = [
    TaskAddParameter(
        id='task1',
        command_line='python script1.py',
        resource_files=[ResourceFile(http_url='https://your_blob_storage_url/script1.py', file_path='script1.py')]
    ),
    TaskAddParameter(
        id='task2',
        command_line='python script2.py',
        resource_files=[ResourceFile(http_url='https://your_blob_storage_url/script2.py', file_path='script2.py')],
        depends_on=TaskDependencies(task_ids=['task1'])
    ),
    TaskAddParameter(
        id='task3',
        command_line='python script3.py',
        resource_files=[ResourceFile(http_url='https://your_blob_storage_url/script3.py', file_path='script3.py')],
        depends_on=TaskDependencies(task_ids=['task2'])
    )
]
# Add tasks to job
batch_client.task.add_collection(job_id, tasks)

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

---

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.