Isolated Worker Durable Function fails with error "Exception while executing function: Did not find any initialized language workers" -- URGENT !!

Rishikesh S Darwade 0

What version of .NET does your existing project use?

.NET 6

What version of .NET are you attempting to target?

.NET 8

Description

After migrating the function app from .NET 6 In-Process to .NET 8 Isolated, we are facing lot of intermittent issues.

Orchestrator and functions are getting timed out. - "Timeout value of 00:20:00 was exceeded by function: Functions.WriteStateActivity"
Exception while executing function: Functions.IntervalCalculatorActivity Did not find any initialized language workers.

Packages :


<PackageReference Include="FlexLabs.EntityFrameworkCore.Upsert" Version="8.0.0" />

<PackageReference Include="Microsoft.Azure.Functions.Extensions" Version="1.1.0" />

<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Timer" Version="4.3.1" />

<PackageReference Include="Microsoft.Azure.WebJobs.Extensions" Version="5.0.0" />

<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.DurableTask" Version="2.9.6" />

<PackageReference Include="Microsoft.EntityFrameworkCore" Version="8.0.0" />

<PackageReference Include="Microsoft.EntityFrameworkCore.SqlServer" Version="8.0.0" />

<PackageReference Include="Microsoft.EntityFrameworkCore.Tools" Version="8.0.0">		<PackageReference Include="Microsoft.Extensions.Configuration.Abstractions" Version="8.0.0" />

		<PackageReference Include="Microsoft.Extensions.DependencyInjection" Version="8.0.0" />

		<PackageReference Include="Oracle.EntityFrameworkCore" Version="8.21.5" />

		<FrameworkReference Include="Microsoft.AspNetCore.App" />

		<PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" Version="1.17.2" />

		<PackageReference Include="Microsoft.ApplicationInsights.WorkerService" Version="2.22.0" />

		<PackageReference Include="Microsoft.Azure.Functions.Worker.ApplicationInsights" Version="1.4.0" />

		<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.DurableTask" Version="1.2.2" />

		<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http" Version="3.2.0" />

		<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore" Version="1.2.1" />

		<PackageReference Include="Microsoft.Azure.Functions.Worker" Version="1.21.0" />

		<PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" Version="1.17.0" />

Rishikesh S Darwade 0

Host JSON file :


```  {

  "version": "2.0",

  "functionTimeout": "00:20:00",

  "extensions": {

    "durableTask": {

      "maxConcurrentActivityFunctions": 25,

      "extendedSessionsEnabled": false

    }

  },

  "logging": {

    "applicationInsights": {

      "samplingSettings": {

        "isEnabled": true,

        "excludedTypes": "Request"

      }

    },

    "logLevel": {

      "Plains.Cts": "Information"

    }

  }

}

Shireesha Eeraboina 1,720 Reputation points Microsoft Vendor

2025-02-07T07:33:14.6233333+00:00
Hi @Rishikesh S Darwade ,

Welcome to the Microsoft Q&A Platform! Thank you for asking your question here.

The error "Did not find any initialized language workers" usually means that the Azure Functions runtime is unable to find the required language workers for the isolated model. Here are some steps you can follow to troubleshoot and resolve this issue:

Ensure that you are using the appropriate versions of the Azure Functions packages. For .NET 8 in the isolated model, you need to replace the Microsoft.Azure.WebJobs.Extensions.DurableTask package with Microsoft.Azure.Functions.Worker.Extensions.DurableTask. The current version of Microsoft.Azure.WebJobs.Extensions.DurableTask you have (2.9.6) is not compatible with the isolated worker model. Update your package reference as follows:
<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.DurableTask" Version="1.2.2" />

Make sure that your host.json file is properly configured. The timeout settings and other configurations should be in accordance with the requirements of the isolated worker model.

The timeout value you have set (00:20:00) is suitable; however, make sure that your orchestrator and activities do not exceed this limit. If they do, consider optimizing the functions to ensure they complete within the allowed time.

If you are migrating and testing, utilize a staging slot to deploy your changes. This enables you to confirm that the application is working correctly before transitioning to production.

Check the logs for any additional errors that could offer more insight into why the language workers are not initializing. Make sure you are using the latest version of the Azure Functions SDK and development tools (such as Visual Studio) to prevent compatibility issues.

For your reference, please review the following documentations for further clarification:

Migrate .NET apps from the in-process model to the isolated worker model

Overview of Durable Functions in the .NET isolated worker

By following these steps, you should be able to address the issues related to the initialization of language workers in your Azure Functions application.

I hope this helps! Let me know if you have any further questions or need additional assistance.
Rishikesh S Darwade 0 Reputation points

2025-02-08T05:39:20.51+00:00
Thanks @Shireesha Eeraboina for the input. The issue related to worker initialization seems to be resolved after replacing the package. However, we are not facing Function Timeout issues sometimes. I have increased the timeout to 30 mins from 20 but still few functions timed out.

This never happens with the previous .NET 6 In-Process version of app. Here is one of the error messages of Orchestrator:

HubName: ctsfunctionapp.

AppName: xxxxx-functionapp.

SlotName: Production.

ExtensionVersion: 3.0.2.

SequenceNumber: 582513.

TaskEventId: 1369

Started

8/2/2025, 9:36:31 am

22041e8b192c4ac78c7914229ee7d27a: Function 'Orchestrator (Orchestrator)' failed with an error. Reason: Microsoft.Azure.WebJobs.Host.FunctionInvocationException at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance, FunctionStartedMessage message, FunctionInstanceLogEntry instanceLogEntry, ParameterHelper parameterHelper, ILogger logger, CancellationToken cancellationToken) in D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 352 at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsync(IFunctionInstance functionInstance, CancellationToken cancellationToken).

Logs at function level.

Timeout value of 00:30:00 was exceeded by function: Functions.IntervalCalculatorActivity
Shireesha Eeraboina 1,720 Reputation points Microsoft Vendor

2025-02-10T04:27:45.99+00:00
Hi @Rishikesh S Darwade ,

I’m glad to hear that the worker initialization issue has been resolved. Regarding the timeout issues you’re experiencing with your functions, here are some steps to help you troubleshoot and potentially resolve the problem:

Examine the logic in your functions, especially in the IntervalCalculatorActivity. Make sure there are no lengthy operations that might be leading to the timeout. It may be beneficial to divide complex tasks into smaller, more manageable functions.

Although you've increased the timeout to 30 minutes, verify that this duration is adequate for the operations being executed. If some tasks naturally require more time, you may need to either extend the timeout further or optimize those tasks.

Make sure you are adhering to best practices for Durable Functions, such as avoiding long-running orchestrations and utilizing durable timers for delays. This approach can help you manage execution time more effectively.

If your functions depend on external services (such as databases or APIs), verify that these services are responsive and not contributing to delays. Network latency or service outages can result in timeouts.

Leverage Application Insights or other logging tools to track the execution of your functions. This can help pinpoint where time is being consumed and whether there are specific patterns contributing to the timeouts.

Make sure that your host.json configuration is compatible with the requirements of the isolated model. The settings should be tailored to optimize performance for your application's specific needs.

If your application is experiencing heavy load, consider scaling out your Function App to accommodate more concurrent executions. This can help reduce the strain on individual function executions.

By following these steps, you should be able to resolve the timeout issues you're facing with your Azure Functions application.

For your reference, please review the following documentations for further clarification:

Overview of Durable Functions in the .NET isolated worker

Migrate .NET apps from the in-process model to the isolated worker model

I hope this helps! Let me know if you have any further questions or need additional assistance.
Rishikesh S Darwade 0 Reputation points

2025-02-10T07:22:22.24+00:00

Thanks @Shireesha Eeraboina for the information. Today I checked the logs and again I can see the error related to language Worker along with timeout.

Exception while executing function: Functions.RetrieveBulletPiDataActivity Did not find any initialized language workers

Type : System.InvalidOperationException Total Count : 46 Message : Did not find any initialized language workers

Here are some more errors related to Runtime captured from Diagnostics:

Exception TypeTotal OccurrencesException MessageLatest TimestampRestart of language worker process(es) completed.57Restart of language worker process(es) completed.2/10/2025 6:08:02 AMRestart of language worker process(es) completed.``57Restart of language worker process(es) completed.2/10/2025 6:08:02 AMA function timeout has occurred. Restarting worker process execu``57A function timeout has occurred. Restarting worker process executing invocationId '8a740fbe-c1a1-456d-9e9f-e602930e3a8a'.2/10/2025 6:08:02 AMSystem.ObjectDisposedException``1Error writing message type InvocationRequest to workerId: 81acd72d-c6b0-4283-ad30-2961617be93f IFeatureCollection has been disposed.2/10/2025 5:36:14 AM
Rishikesh S Darwade 0 Reputation points

2025-02-10T07:22:49.1333333+00:00

Timestamp : 2/10/2025 5:36:14 AM Inner Exception Type: System.ObjectDisposedException Total Occurrences: 1 Latest Exception Message: Error writing message type InvocationRequest to workerId: 81acd72d-c6b0-4283-ad30-2961617be93f IFeatureCollection has been disposed. Object name: 'Collection'.

Full Exception : System.ObjectDisposedException : IFeatureCollection has been disposed. Object name: 'Collection'. at Microsoft.AspNetCore.Http.Features.FeatureReferences1.ThrowContextDisposed() at Microsoft.AspNetCore.Http.Features.FeatureReferences1.Fetch[TFeature,TState] at Microsoft.AspNetCore.Http.Features.FeatureReferences1.Fetch[TFeature](TFeature& cached,Func2 factory) at Microsoft.AspNetCore.Http.DefaultHttpContext.get_RequestAborted() at Grpc.AspNetCore.Server.Internal.HttpContextServerCallContext.get_CancellationTokenCore() at Grpc.Core.ServerCallContext.get_CancellationToken() at async Grpc.AspNetCore.Server.Internal.PipeExtensions.WriteStreamedMessageAsync[TResponse] at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Grpc.AspNetCore.Server.Internal.HttpContextStreamWriter1.WriteCoreAsync[TResponse](TResponse message,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Script.Grpc.FunctionRpcService.PushFromOutboundToGrpc(String workerId,IServerStreamWriter1 responseStream,ChannelReader1 source,CancellationToken cancellationToken) at /_/src/WebJobs.Script.Grpc/Server/FunctionRpcService.cs : 110 Full Exception : System.ObjectDisposedException : IFeatureCollection has been disposed. Object name: 'Collection'. at Microsoft.AspNetCore.Http.Features.FeatureReferences1.ThrowContextDisposed() at Microsoft.AspNetCore.Http.Features.FeatureReferences1.Fetch[TFeature,TState](TFeature& cached,TState state,Func2 factory) at Microsoft.AspNetCore.Http.Features.FeatureReferences1.Fetch[TFeature](TFeature& cached,Func2 factory) at Microsoft.AspNetCore.Http.DefaultHttpContext.get_RequestAborted() at Grpc.AspNetCore.Server.Internal.HttpContextServerCallContext.get_CancellationTokenCore() at Grpc.Core.ServerCallContext.get_CancellationToken() at async Grpc.AspNetCore.Server.Internal.PipeExtensions.WriteStreamedMessageAsync[TResponse] at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Grpc.AspNetCore.Server.Internal.HttpContextStreamWriter1.WriteCoreAsync[TResponse](TResponse message,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Script.Grpc.FunctionRpcService.PushFromOutboundToGrpc(String workerId,IServerStreamWriter1 responseStream,ChannelReader`1 source,CancellationToken cancellationToken) at /_/src/WebJobs.Script.Grpc/Server/FunctionRpcService.cs : 110
Shireesha Eeraboina 1,720 Reputation points Microsoft Vendor

2025-02-11T07:02:00.1966667+00:00
Hi @Rishikesh S Darwade ,

Thanks for sharing the Additional details from these logs I understand that post migrating to .net 8 runtime is not able to find the language to load.

Type: System.InvalidOperationException Total Count: 46 Message: Did not find any initialized language workers Here are some more errors related to Runtime captured from Diagnostics:

So for the above i request you to please add this setting "FUNCTIONS_WORKER_RUNTIME": "dotnet-isolated", in the host.json file here is the reference doc for the format of that setting : host.json reference for Azure Functions 2.x | Microsoft Learn

Also, if you are deploying the function from visual studio or from local add the above setting in your local.settings.json file.

And see whether the runtime is able to load the functions or not
Rishikesh S Darwade 0 Reputation points

2025-02-11T08:38:07.0366667+00:00

Hi @Shireesha Eeraboina , thanks for the information. I checked the ref doc that you shared but I am not able to find the setting FUNCTIONS_WORKER_RUNTIME in host.json file.

I have this setting in the Environment Variables of the deployed function app and the deployment is happening through Release pipeline.

While running on local, I have the required setting FUNCTIONS_WORKER_RUNTIME in local.settings.json file.

Could you please share the format how to add this setting in the host.json ?
Shireesha Eeraboina 1,720 Reputation points Microsoft Vendor

2025-02-12T13:37:17.0966667+00:00
Hi @Rishikesh S Darwade ,

I understand that you're looking to add the FUNCTIONS_WORKER_RUNTIME setting to your host.json file. However, this specific setting is not typically included in the host.json file; instead, it should be set as an environment variable for your Azure Function App.

Since you mentioned that you already have this setting in the Environment Variables of your deployed function app, that is indeed the correct approach. The host.json file is primarily used for configuration settings related to the function's behavior, while environment variables are used to specify the runtime.

To clarify, please keep the FUNCTIONS_WORKER_RUNTIME setting in your Environment Variables for the deployed function app. Additionally, ensure that it is also present in your local.settings.json file for local development.

If you need to verify or modify the environment variables in Azure, you can do so through the Azure Portal:

Navigate to your Function App.

Go to the "Configuration" section.

Under "Application settings," you can add or edit the FUNCTIONS_WORKER_RUNTIME variable.

For your reference follow the below screenshot:

If you have any further questions or need additional assistance, feel free to ask!
Rishikesh S Darwade 0 Reputation points

2025-02-12T13:55:12.9033333+00:00

Hi @Shireesha Eeraboina , You are correct that the FUNCTIONS_WORKER_RUNTIME setting is part of the Configuration. However, in your previous reply, you suggested adding it to the host.json file, which is why I wanted to clarify this point.

That said, I am still encountering the Did not find any initialized language workers error, even though the Function Worker Runtime is set correctly. Unfortunately, I haven’t been able to resolve this issue yet.

Could you please assist in expediting this matter as it is urgent?
Shireesha Eeraboina 1,720 Reputation points Microsoft Vendor

2025-02-17T01:32:52.16+00:00

Hi @Rishikesh S Darwade ,

Just checking in to see if the information above was helpful. If you have any further updates on this issue, please feel free to post them here.

1 answer

Shireesha Eeraboina 1,720 Reputation points Microsoft Vendor

2025-02-13T16:43:43.7433333+00:00
Hi @Rishikesh S Darwade ,

The error "Did not find any initialized language workers" suggests that there may be an issue with the function app not being able to properly initialize the language workers (in this case, for .NET 8 Isolated).

Possible Causes & Solutions:

Worker Initialization Failure: The isolated model requires a different worker setup. It might be failing to start the required .NET worker for your function. This could happen due to missing dependencies, incorrect configuration, or insufficient resources.

Solution: Verify that all the required dependencies are correctly installed and configured in the function app. Ensure that the .NET 8 SDK and runtime are available in the environment.

Function App Configuration:

Double-check the settings in the host.json file for the isolated worker, especially for version compatibility. Ensure you are using the correct settings for .NET 8 Isolated.

Example of host.json configuration for Isolated Worker:

Cold Start Issues: The isolated model may cause the application to experience cold starts or delayed initialization due to the isolated worker setup. You can reduce cold starts by keeping the app "warm" or adjusting the scale settings.

Solution: Enable Always On if using a Premium or App Service plan. This prevents your function app from going idle and might help mitigate cold start issues.

For your reference, please review the following documentations for further clarification:

Supported languages in Azure Functions | Microsoft Learn

I hope this answers your query! Let me know if you require any additional help or clarification.
Please sign in to rate this answer.
Shireesha Eeraboina 1,720 Reputation points Microsoft Vendor

2025-02-18T04:48:40.5033333+00:00

Hi @Rishikesh S Darwade ,

If you found the response helpful, please click Accept Answer and Yes for the provided answer. This will help other community members with similar issues find the solution more easily.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Isolated Worker Durable Function fails with error "Exception while executing function: Did not find any initialized language workers" -- URGENT !!

1 answer

Your answer