I wrote an Azure Function App with Durable Functions. While testing with a large dataset, I realized that the functionTimeout value was 10 mins. This caused some of the orchestrations to timeout. Thus began an indefinite number of retries of the failed orchestrations. I have increased the timeout setting to 1 hour 30 mins, but some of the old retries are still pending and therefore still running. I have tried stopping the function app and restarting it back up, but the execution of the orchestrations continued. I have also tried constructing the terminatePostUri query string but I see an HTTP 404 error. Two questions: How do I stop the execution of the orchestrations? How do I set the number of retries of the orchestrations (not the activity calls) to 0?

@Anil Bhat To stop the execution of the orchestrations, you can use the TerminateAsync method provided by the IDurableOrchestrationClient interface. This method allows you to terminate a running orchestration instance. Here's an example of how you can use it: [FunctionName("TerminateOrchestration")] public static async Task<IActionResult> TerminateOrchestration( [HttpTrigger(AuthorizationLevel.Function, "post", Route = null)] HttpRequest req, [DurableClient] IDurableOrchestrationClient client, ILogger log) { string instanceId = req.Query["instanceId"]; string reason = req.Query["reason"]; await client.TerminateAsync(instanceId, reason); return new OkResult(); } You can call this function with the instance ID of the orchestration you want to terminate and a reason for the termination. To set the number of retries of the orchestrations to 0, you can configure the retry policy for the orchestrator function. By default, orchestrator functions do not retry on failure. However, if you have configured a retry policy, you can set the maximum number of attempts to 1 to effectively disable retries. Here's an example of how you can configure the retry policy: var retryOptions = new RetryOptions( firstRetryInterval: TimeSpan.FromSeconds(5), maxNumberOfAttempts: 1 // Set to 1 to disable retries ); await context.CallActivityWithRetryAsync("ActivityFunctionName", retryOptions, input); This will ensure that the orchestrator function does not retry on failure. I hope this helps. Let me know if you have any other questions. See: https://github.com/Azure/azure-functions-durable-extension/discussions/2029 https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-instance-management?tabs=csharp https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-orchestrations?tabs=csharp-inproc https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-sub-orchestrations?tabs=csharp-inproc

I need to avoid indefinite number of retries of durable function orchestrations

Accepted answer

Chiugo Okpala 545 Reputation points MVP

2025-02-19T05:19:20.3833333+00:00
@Anil Bhat

To stop the execution of the orchestrations, you can use the TerminateAsync method provided by the IDurableOrchestrationClient interface. This method allows you to terminate a running orchestration instance. Here's an example of how you can use it:

[FunctionName("TerminateOrchestration")] public static async Task<IActionResult> TerminateOrchestration( [HttpTrigger(AuthorizationLevel.Function, "post", Route = null)] HttpRequest req, [DurableClient] IDurableOrchestrationClient client, ILogger log) { string instanceId = req.Query["instanceId"]; string reason = req.Query["reason"]; await client.TerminateAsync(instanceId, reason); return new OkResult(); }

You can call this function with the instance ID of the orchestration you want to terminate and a reason for the termination.

To set the number of retries of the orchestrations to 0, you can configure the retry policy for the orchestrator function. By default, orchestrator functions do not retry on failure. However, if you have configured a retry policy, you can set the maximum number of attempts to 1 to effectively disable retries. Here's an example of how you can configure the retry policy:

var retryOptions = new RetryOptions( firstRetryInterval: TimeSpan.FromSeconds(5), maxNumberOfAttempts: 1 // Set to 1 to disable retries ); await context.CallActivityWithRetryAsync("ActivityFunctionName", retryOptions, input);

This will ensure that the orchestrator function does not retry on failure.

I hope this helps. Let me know if you have any other questions.

See:

https://github.com/Azure/azure-functions-durable-extension/discussions/2029

https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-instance-management?tabs=csharp

https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-orchestrations?tabs=csharp-inproc

https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-sub-orchestrations?tabs=csharp-inproc
Please sign in to rate this answer.
Anil Bhat 20 Reputation points

2025-02-19T15:24:53.1433333+00:00

Hello Chiugo,

I would like to terminate the orchestrator instance without using code. What would a terminatePostUri look like? I tried the following: https://<appname>.azurewebsites.net/runtime/webhooks/durableTask/instances/<instance id>/terminate?code=<system key>. However, I saw an HTTP 404 error.

I have not set any retry value and yet the orchestration restarts upon failure. In addition, the code you provided applies to an activity called by an orchestration, not the orchestration itself. Am I reading it wrong?

Chiugo Okpala 545 Reputation points MVP

2025-02-19T16:09:34.2933333+00:00

Hi Anil,

To terminate an orchestrator instance without using code, you can use the HTTP API provided by Azure Durable Functions. The terminatePostUri should look like this:

https://<appname>.azurewebsites.net/runtime/webhooks/durableTask/instances/<instanceId>/terminate?reason=<reason>&code=<systemKey>

Make sure to replace <appname>, <instanceId>, <reason>, and <systemKey> with your actual values. If you are seeing an HTTP 404 error, it might be due to an incorrect instance ID or system key. Double-check these values to ensure they are correct.

Regarding the retries, if you have not set any retry policy and the orchestration is still restarting upon failure, it might be due to the default behavior of the Durable Task Framework. By default, orchestrator functions do not retry on failure, but activity functions can have retry policies. If you want to ensure that the orchestrator itself does not retry, you can explicitly set the retry policy for the orchestrator function to have zero retries.

Here's an example of how you can configure the retry policy for the orchestrator function:

var retryOptions = new RetryOptions( firstRetryInterval: TimeSpan.FromSeconds(5), maxNumberOfAttempts: 1 // Set to 1 to disable retries ); await context.CallActivityWithRetryAsync("ActivityFunctionName", retryOptions, input);

This will ensure that the orchestrator function does not retry on failure.

For more details, you can refer to the Azure Durable Functions HTTP API documentation.

I hope this helps. Let me know if you have any other questions.

Anil Bhat 20 Reputation points

2025-02-21T03:29:40.97+00:00

I eventually decided to delete the function app and the associated resources, and redeploy them. By the way, my function app is written in Python.

Two questions:

What does CallActivityWithRetryAsync have to do with the orchestration? Is there an equivalent method for the orchestrator itself?

Does the termination code need to be written in a separate orchestration?

Chiugo Okpala 545 Reputation points MVP

2025-02-21T06:11:06.28+00:00

@Anil Bhat

Great to hear you found a solution by redeploying your function app. Let's address your questions:

1. What does CallActivityWithRetryAsync have to do with the orchestration? Is there an equivalent method for the orchestrator itself?

The CallActivityWithRetryAsync method is used within an orchestration to call activity functions with retry logic. This means that if an activity function fails, it will automatically retry according to the specified retry policy. This method is specific to activities within the orchestration.

For the orchestration itself, there isn't a direct equivalent method because orchestrations are designed to manage the overall workflow and state transitions, rather than handling retry logic themselves. Instead, retries at the orchestration level are managed through the configuration in the host.json file, as mentioned earlier.

2. Does the termination code need to be written in a separate orchestration?

No, the termination code doesn't need to be written in a separate orchestration. You can use the HTTP API to terminate specific orchestrations without writing additional code. Just make sure the terminate URL is correctly formatted and you have the correct instance ID and system key.

Here’s a reminder of the correct terminate URL format:

https://<appname>.azurewebsites.net/runtime/webhooks/durabletask/instances/<instance-id>/terminate?code=<system-key>

Ensure that you replace <appname>, <instance-id>, and <system-key> with the appropriate values for your function app.

If you face issues with the HTTP 404 error, double-check the instance ID and system key for correctness.

Using Python for Azure Durable Functions

Since your function app is written in Python, here’s an example of how you might use call_activity_with_retry for activity functions:

import azure.durable_functions as df def orchestrator_function(context: df.DurableOrchestrationContext): retry_options = df.RetryOptions(1000, 3) # 1 second interval, 3 retries result = yield context.call_activity_with_retry('MyActivity', retry_options, input_data) return result main = df.Orchestrator.create(orchestrator_function)

This example shows how you can use retry options for activities within your Python orchestration. While there isn't a direct retry method for the orchestration itself, you can configure the retry policy through the host.json as mentioned previously.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

I need to avoid indefinite number of retries of durable function orchestrations

0 additional answers

Your answer