Azure batch immediately returning trigger to dependent task when resourcefilles are added

Question

Hi,

I am using Azure batch task dependencies feature and facing one issue. I have one parent task which needs some data to process. I have configured ResourceFiles for my parent task so that it can download the files to process. I have dependent (child) task which should only trigger when parent task has completed it's process successfully (i.e., State = Success).

However, as ResourceFiles is configured for Parent task, it is immediately triggering the dependent (child) task once all files are downloaded via ResouceFiles from parent task. Which is not what I want. My dependent task should only run when parent task has executed the CommandLine and stored the Output file (which will be used in child task).

Code:

await CreateBatchPoolIfNotExists(BatchAccountResourceID, poolId);

List<CloudTask> tasks = new List<CloudTask>()
{
    new CloudTask("Parent","cmd /c [ffmpeg command]")
    {
        ResourceFiles = new List<ResourceFile>() 
        {
           ResourceFile.FromStorageContainerUrl("[CONTAINER_SAS_URL]", blobPrefix: "[BLOB_PATH_PREFIX]")
        },
        OutputFiles = new List<OutputFile>()
        {
            new OutputFile(
               filePattern: @"output_*.mp4",
               destination:
            new OutputFileDestination(new OutputFileBlobContainerDestination(
                       containerUrl: "[CONTAINER_SAS_URL]",
                       path: "[OUTPUT_BLOB_PATH]")),
               uploadOptions: new OutputFileUploadOptions(
               uploadCondition: OutputFileUploadCondition.TaskSuccess))
        }
    },
    new CloudTask("Child","cmd /c [ffmpeg command]")
    {
        DependsOn = TaskDependencies.OnId("Parent"),
        ResourceFiles = new List<ResourceFile>()
        {
           ResourceFile.FromStorageContainerUrl("[CONTAINER_SAS_URL]", blobPrefix: "[OUTPUT_BLOB_PATH_PREFIX]")
        },
        OutputFiles = new List<OutputFile>()
        {
            new OutputFile(
               filePattern: @"result_*.mp4",
               destination:
            new OutputFileDestination(new OutputFileBlobContainerDestination(
                       containerUrl: "[CONTAINER_SAS_URL]",
                       path: "[RESULT_BLOB_PATH]")),
               uploadOptions: new OutputFileUploadOptions(
               uploadCondition: OutputFileUploadCondition.TaskSuccess))
        }
    },
};

batchClient.JobOperations.AddTask(JobId, tasks);

Above code is executing successfully and able to create task and it's dependencies. However it end up triggering Child task without waiting for Parent to complete it's process completely (until Output is not created).

Please suggest the way forward.

Answer 1

Hi Parikh, Disham [ext],

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others " Accept an answer on Microsoft Q&A, I'll repost your solution in case you'd like to "Accept " the answer.

Issue:

Azure batch immediately returning trigger to dependent task when resourcefulness are added.

Solution:

I was using same task name every time. so probably Microsoft is keeping some cached value inside a job and when I remove previous task and create a new Task with the name (Parent and Child), the state of the parent task was being fetched from the cache (assumption), and it was triggering the child task immediately.

'Every time create a task with unique name to avoid being in this situation'

If you have any other questions or are still running into more issues, please let me know. Thank you again for your time and patience throughout this issue.

Please remember to "Accept Answer" if any answer/reply helped, so that others in the community facing similar issues can easily find the solution.

Thank you.

Answer 2

Could you please verify if you have the following snippet in your code prior to creating the list of tasks for execution :

// IMPORTANT: This is REQUIRED for using task dependencies.

unboundJob.UsesTaskDependencies = true;

await unboundJob.CommitAsync();

If not, please add and then can you post if that works?