How can we avoid non-deterministic errors when changing the task sequence in an orchestrator?

Question

I am encountering a non-deterministic workflow error in my Azure Durable Function orchestration. The error occurs when an orchestration replay schedules a different activity task than expected, causing a mismatch in sequence numbers. My orchestration appears to be attempting to replay a previously executed task but is failing due to a code change or a mismatch in scheduled activity names and sequence numbers. I’d like to understand the root cause of this issue and get guidance on best practices to prevent non-deterministic errors when modifying orchestrator code.

I recently updated my orchestrator code, renaming or reordering certain activity functions. Any insights on handling code changes without breaking orchestrator determinism or tips on debugging this issue would be greatly appreciated.

Thank you!

Answer 1

Hello Muhammed , Welcome to MS Q&A

Non-deterministic errors in Azure Durable Functions orchestrator code can arise from several factors, primarily due to the nature of how orchestrator functions are designed to work. The root cause of these errors is often related to the use of non-deterministic constructs, such as static variables, environment variables, or any form of state that can change between executions.

To prevent non-deterministic errors when modifying orchestrator code, consider the following best practices:

1. Avoid Static Variables: Static variables can lead to unpredictable behavior since their values may change over time. Instead, use constants or limit static variable usage to activity functions.
2. Do Not Use Environment Variables: Environment variables can also change, leading to non-deterministic behavior. If configuration values are needed, pass them into the orchestrator function as inputs or return them from activity functions.
3. Use Activity Functions for Outbound Calls: Any outbound network calls should be made from activity functions rather than orchestrator functions. This helps maintain the determinism of orchestrator functions.
4. Avoid Blocking APIs: Blocking calls, such as sleep, can cause performance issues and should be avoided. Use Durable timers instead for creating delays.
5. Handle Asynchronous Operations Properly: Orchestrator functions should not start any async operations outside of those defined by the orchestration trigger's context. Use Durable SDK APIs for scheduling async work.
6. Implement Unique Identifiers for External Events: Since external events have an at-least-once delivery guarantee, including unique IDs in these events can help manage duplicates effectively.

By adhering to these practices, you can enhance the reliability and determinism of your orchestrator functions, reducing the likelihood of encountering non-deterministic errors.

References:

• Durable orchestrations
• Orchestrator function code constraints
• Durable Functions best practices and diagnostic tool

Please let us know if you have any further questions

Kindly accept answer if it helps

Thanks

Deepanshu