Isolated Worker Durable Function fails with error "Exception while executing function: Did not find any initialized language workers" -- URGENT !!

Rishikesh S Darwade 0

What version of .NET does your existing project use?

.NET 6

What version of .NET are you attempting to target?

.NET 8

Description

After migrating the function app from .NET 6 In-Process to .NET 8 Isolated, we are facing lot of intermittent issues.

Orchestrator and functions are getting timed out. - "Timeout value of 00:20:00 was exceeded by function: Functions.WriteStateActivity"
Exception while executing function: Functions.IntervalCalculatorActivity Did not find any initialized language workers.

Packages :


<PackageReference Include="FlexLabs.EntityFrameworkCore.Upsert" Version="8.0.0" />

<PackageReference Include="Microsoft.Azure.Functions.Extensions" Version="1.1.0" />

<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Timer" Version="4.3.1" />

<PackageReference Include="Microsoft.Azure.WebJobs.Extensions" Version="5.0.0" />

<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.DurableTask" Version="2.9.6" />

<PackageReference Include="Microsoft.EntityFrameworkCore" Version="8.0.0" />

<PackageReference Include="Microsoft.EntityFrameworkCore.SqlServer" Version="8.0.0" />

<PackageReference Include="Microsoft.EntityFrameworkCore.Tools" Version="8.0.0">		<PackageReference Include="Microsoft.Extensions.Configuration.Abstractions" Version="8.0.0" />

		<PackageReference Include="Microsoft.Extensions.DependencyInjection" Version="8.0.0" />

		<PackageReference Include="Oracle.EntityFrameworkCore" Version="8.21.5" />

		<FrameworkReference Include="Microsoft.AspNetCore.App" />

		<PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" Version="1.17.2" />

		<PackageReference Include="Microsoft.ApplicationInsights.WorkerService" Version="2.22.0" />

		<PackageReference Include="Microsoft.Azure.Functions.Worker.ApplicationInsights" Version="1.4.0" />

		<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.DurableTask" Version="1.2.2" />

		<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http" Version="3.2.0" />

		<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore" Version="1.2.1" />

		<PackageReference Include="Microsoft.Azure.Functions.Worker" Version="1.21.0" />

		<PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" Version="1.17.0" />

Rishikesh S Darwade 0

Host JSON file :


```  {

  "version": "2.0",

  "functionTimeout": "00:20:00",

  "extensions": {

    "durableTask": {

      "maxConcurrentActivityFunctions": 25,

      "extendedSessionsEnabled": false

    }

  },

  "logging": {

    "applicationInsights": {

      "samplingSettings": {

        "isEnabled": true,

        "excludedTypes": "Request"

      }

    },

    "logLevel": {

      "Plains.Cts": "Information"

    }

  }

}

Shireesha Eeraboina 1,635 Reputation points Microsoft Vendor

2025-02-07T07:33:14.6233333+00:00
Hi @Rishikesh S Darwade ,

Welcome to the Microsoft Q&A Platform! Thank you for asking your question here.

The error "Did not find any initialized language workers" usually means that the Azure Functions runtime is unable to find the required language workers for the isolated model. Here are some steps you can follow to troubleshoot and resolve this issue:

Ensure that you are using the appropriate versions of the Azure Functions packages. For .NET 8 in the isolated model, you need to replace the Microsoft.Azure.WebJobs.Extensions.DurableTask package with Microsoft.Azure.Functions.Worker.Extensions.DurableTask. The current version of Microsoft.Azure.WebJobs.Extensions.DurableTask you have (2.9.6) is not compatible with the isolated worker model. Update your package reference as follows:
<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.DurableTask" Version="1.2.2" />

Make sure that your host.json file is properly configured. The timeout settings and other configurations should be in accordance with the requirements of the isolated worker model.

The timeout value you have set (00:20:00) is suitable; however, make sure that your orchestrator and activities do not exceed this limit. If they do, consider optimizing the functions to ensure they complete within the allowed time.

If you are migrating and testing, utilize a staging slot to deploy your changes. This enables you to confirm that the application is working correctly before transitioning to production.

Check the logs for any additional errors that could offer more insight into why the language workers are not initializing. Make sure you are using the latest version of the Azure Functions SDK and development tools (such as Visual Studio) to prevent compatibility issues.

For your reference, please review the following documentations for further clarification:

Migrate .NET apps from the in-process model to the isolated worker model

Overview of Durable Functions in the .NET isolated worker

By following these steps, you should be able to address the issues related to the initialization of language workers in your Azure Functions application.

I hope this helps! Let me know if you have any further questions or need additional assistance.
Rishikesh S Darwade 0 Reputation points

2025-02-08T05:39:20.51+00:00
Thanks @Shireesha Eeraboina for the input. The issue related to worker initialization seems to be resolved after replacing the package. However, we are not facing Function Timeout issues sometimes. I have increased the timeout to 30 mins from 20 but still few functions timed out.

This never happens with the previous .NET 6 In-Process version of app. Here is one of the error messages of Orchestrator:

HubName: ctsfunctionapp.

AppName: xxxxx-functionapp.

SlotName: Production.

ExtensionVersion: 3.0.2.

SequenceNumber: 582513.

TaskEventId: 1369

Started

8/2/2025, 9:36:31 am

22041e8b192c4ac78c7914229ee7d27a: Function 'Orchestrator (Orchestrator)' failed with an error. Reason: Microsoft.Azure.WebJobs.Host.FunctionInvocationException at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance, FunctionStartedMessage message, FunctionInstanceLogEntry instanceLogEntry, ParameterHelper parameterHelper, ILogger logger, CancellationToken cancellationToken) in D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 352 at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsync(IFunctionInstance functionInstance, CancellationToken cancellationToken).

Logs at function level.

Timeout value of 00:30:00 was exceeded by function: Functions.IntervalCalculatorActivity
Shireesha Eeraboina 1,635 Reputation points Microsoft Vendor

2025-02-10T04:27:45.99+00:00
Hi @Rishikesh S Darwade ,

I’m glad to hear that the worker initialization issue has been resolved. Regarding the timeout issues you’re experiencing with your functions, here are some steps to help you troubleshoot and potentially resolve the problem:

Examine the logic in your functions, especially in the IntervalCalculatorActivity. Make sure there are no lengthy operations that might be leading to the timeout. It may be beneficial to divide complex tasks into smaller, more manageable functions.

Although you've increased the timeout to 30 minutes, verify that this duration is adequate for the operations being executed. If some tasks naturally require more time, you may need to either extend the timeout further or optimize those tasks.

Make sure you are adhering to best practices for Durable Functions, such as avoiding long-running orchestrations and utilizing durable timers for delays. This approach can help you manage execution time more effectively.

If your functions depend on external services (such as databases or APIs), verify that these services are responsive and not contributing to delays. Network latency or service outages can result in timeouts.

Leverage Application Insights or other logging tools to track the execution of your functions. This can help pinpoint where time is being consumed and whether there are specific patterns contributing to the timeouts.

Make sure that your host.json configuration is compatible with the requirements of the isolated model. The settings should be tailored to optimize performance for your application's specific needs.

If your application is experiencing heavy load, consider scaling out your Function App to accommodate more concurrent executions. This can help reduce the strain on individual function executions.

By following these steps, you should be able to resolve the timeout issues you're facing with your Azure Functions application.

For your reference, please review the following documentations for further clarification:

Overview of Durable Functions in the .NET isolated worker

Migrate .NET apps from the in-process model to the isolated worker model

I hope this helps! Let me know if you have any further questions or need additional assistance.
Rishikesh S Darwade 0 Reputation points

2025-02-10T07:22:22.24+00:00

Thanks @Shireesha Eeraboina for the information. Today I checked the logs and again I can see the error related to language Worker along with timeout.

Exception while executing function: Functions.RetrieveBulletPiDataActivity Did not find any initialized language workers

Type : System.InvalidOperationException Total Count : 46 Message : Did not find any initialized language workers

Here are some more errors related to Runtime captured from Diagnostics:

Exception TypeTotal OccurrencesException MessageLatest TimestampRestart of language worker process(es) completed.57Restart of language worker process(es) completed.2/10/2025 6:08:02 AMRestart of language worker process(es) completed.``57Restart of language worker process(es) completed.2/10/2025 6:08:02 AMA function timeout has occurred. Restarting worker process execu``57A function timeout has occurred. Restarting worker process executing invocationId '8a740fbe-c1a1-456d-9e9f-e602930e3a8a'.2/10/2025 6:08:02 AMSystem.ObjectDisposedException``1Error writing message type InvocationRequest to workerId: 81acd72d-c6b0-4283-ad30-2961617be93f IFeatureCollection has been disposed.2/10/2025 5:36:14 AM
Rishikesh S Darwade 0 Reputation points

2025-02-10T07:22:49.1333333+00:00

Timestamp : 2/10/2025 5:36:14 AM Inner Exception Type: System.ObjectDisposedException Total Occurrences: 1 Latest Exception Message: Error writing message type InvocationRequest to workerId: 81acd72d-c6b0-4283-ad30-2961617be93f IFeatureCollection has been disposed. Object name: 'Collection'.

Full Exception : System.ObjectDisposedException : IFeatureCollection has been disposed. Object name: 'Collection'. at Microsoft.AspNetCore.Http.Features.FeatureReferences1.ThrowContextDisposed() at Microsoft.AspNetCore.Http.Features.FeatureReferences1.Fetch[TFeature,TState] at Microsoft.AspNetCore.Http.Features.FeatureReferences1.Fetch[TFeature](TFeature& cached,Func2 factory) at Microsoft.AspNetCore.Http.DefaultHttpContext.get_RequestAborted() at Grpc.AspNetCore.Server.Internal.HttpContextServerCallContext.get_CancellationTokenCore() at Grpc.Core.ServerCallContext.get_CancellationToken() at async Grpc.AspNetCore.Server.Internal.PipeExtensions.WriteStreamedMessageAsync[TResponse] at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Grpc.AspNetCore.Server.Internal.HttpContextStreamWriter1.WriteCoreAsync[TResponse](TResponse message,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Script.Grpc.FunctionRpcService.PushFromOutboundToGrpc(String workerId,IServerStreamWriter1 responseStream,ChannelReader1 source,CancellationToken cancellationToken) at /_/src/WebJobs.Script.Grpc/Server/FunctionRpcService.cs : 110 Full Exception : System.ObjectDisposedException : IFeatureCollection has been disposed. Object name: 'Collection'. at Microsoft.AspNetCore.Http.Features.FeatureReferences1.ThrowContextDisposed() at Microsoft.AspNetCore.Http.Features.FeatureReferences1.Fetch[TFeature,TState](TFeature& cached,TState state,Func2 factory) at Microsoft.AspNetCore.Http.Features.FeatureReferences1.Fetch[TFeature](TFeature& cached,Func2 factory) at Microsoft.AspNetCore.Http.DefaultHttpContext.get_RequestAborted() at Grpc.AspNetCore.Server.Internal.HttpContextServerCallContext.get_CancellationTokenCore() at Grpc.Core.ServerCallContext.get_CancellationToken() at async Grpc.AspNetCore.Server.Internal.PipeExtensions.WriteStreamedMessageAsync[TResponse] at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Grpc.AspNetCore.Server.Internal.HttpContextStreamWriter1.WriteCoreAsync[TResponse](TResponse message,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Script.Grpc.FunctionRpcService.PushFromOutboundToGrpc(String workerId,IServerStreamWriter1 responseStream,ChannelReader`1 source,CancellationToken cancellationToken) at /_/src/WebJobs.Script.Grpc/Server/FunctionRpcService.cs : 110
Shireesha Eeraboina 1,635 Reputation points Microsoft Vendor

2025-02-11T07:02:00.1966667+00:00
Hi @Rishikesh S Darwade ,

Thanks for sharing the Additional details from these logs I understand that post migrating to .net 8 runtime is not able to find the language to load.

Type: System.InvalidOperationException Total Count: 46 Message: Did not find any initialized language workers Here are some more errors related to Runtime captured from Diagnostics:

So for the above i request you to please add this setting "FUNCTIONS_WORKER_RUNTIME": "dotnet-isolated", in the host.json file here is the reference doc for the format of that setting : host.json reference for Azure Functions 2.x | Microsoft Learn

Also, if you are deploying the function from visual studio or from local add the above setting in your local.settings.json file.

And see whether the runtime is able to load the functions or not
Rishikesh S Darwade 0 Reputation points

2025-02-11T08:38:07.0366667+00:00

Hi @Shireesha Eeraboina , thanks for the information. I checked the ref doc that you shared but I am not able to find the setting FUNCTIONS_WORKER_RUNTIME in host.json file.

I have this setting in the Environment Variables of the deployed function app and the deployment is happening through Release pipeline.

While running on local, I have the required setting FUNCTIONS_WORKER_RUNTIME in local.settings.json file.

Could you please share the format how to add this setting in the host.json ?

Share via

Isolated Worker Durable Function fails with error "Exception while executing function: Did not find any initialized language workers" -- URGENT !!

Your answer