ADF Pipeline Fails with Error ‘StatusCode: DFExecutorUserError, Message: Job Failed Due to Reason: Failed to Create a Child Event Loop’

Question

Hi team,

My ADF pipeline recently started failing with the following error. My pipeline takes data from Cosmos DB, transforms it, and dumps it as a CSV file into a storage account.

Error Message:

Operation on target Data_Transfer failed: {"StatusCode":"DFExecutorUserError","Message":"Job failed due to reason: failed to create a child event loop","Details":"java.lang.IllegalStateException: failed to create a child event loop\n\tat shaded.msdataflowtest.azurecosmosspark.io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:88)\n\tat shaded.msdataflowtest.azurecosmosspark.io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:60)\n\tat shaded.msdataflowtest.azurecosmosspark.io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:49)\n\tat shaded.msdataflowtest.azurecosmosspark.io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59)\n\tat shaded.msdataflowtest.azurecosmosspark.io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:87)\n\tat shaded.msdataflowtest.azurecosmosspark.io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:82)\n\tat shaded.msdataflowtest.azurecosmosspark.io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:69)\n\tat shaded.msdataflowtest.azure_cosmos_spark.co"}

The pipeline was working for 3 months without any issues until 5 days ago. I recently started using system-assigned managed identity to access Cosmos DB via linked service, and I suspect the issue started since then.

Additionally, although the pipeline sometimes succeeds, it does not write anything to the target.

Please help me identify the issue and provide a solution.