InvalidHttpRequestToLivy: Submission failed due to error content =["requirement failed: Session isn't active."] HTTP status code: 400.

Daniel Wang 0

I searched the web high and low, and cannot find a viable solution.

I am using Azure Synapse.

I have 24 files within the Gen2 storage account, which I run a loop to read_csv for them all. Then write them into delta table with each run.

After around 40 mins or so, "InvalidHttpRequestToLivy: Submission failed due to error content =["requirement failed: Session isn't active."] HTTP status code: 400." message pops up.

My company is using the lowest tier with limited nodes and cores. How do I extend the time for this?

phemanth 12,740 Reputation points Microsoft Vendor

2024-12-30T18:43:27.38+00:00
@Daniel Wang

Welcome to the Microsoft Q&A and thank you for posting your questions here.

Please check the below steps and confirm us

Increase the Session Timeout: You can extend the session timeout by setting the livy.server.session.timeout property in your Spark configuration. For example, you can set it to 4 hours:

spark.conf.set("livy.server.session.timeout", "4h")

This should give your long-running jobs more time to complete

Optimize Your Spark Job: Try to optimize your Spark job to reduce execution time. This can include:

Improving the efficiency of your Spark code.

Tuning Spark configurations.

Increasing the resources allocated to your Spark job1.

Check Network Connectivity: Ensure there are no network connectivity issues between the Livy server and the Spark driver. This includes checking firewall and network settings

Use a Different Spark Pool: If possible, try running your Spark job on a different Spark pool to see if that resolves the issue

Review Spark Logs: Check the Spark logs for any errors or warnings that might be causing the issue. You can access these logs from the Azure Synapse Analytics workspace or the Azure portal

I hope the above steps will resolve the issue, please do let us know if issue persists. Thank you
phemanth 12,740 Reputation points Microsoft Vendor

2024-12-31T18:08:31.5166667+00:00

@Daniel Wang We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
phemanth 12,740 Reputation points Microsoft Vendor

2025-01-02T16:40:03.68+00:00

@Daniel Wang just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Share via

InvalidHttpRequestToLivy: Submission failed due to error content =["requirement failed: Session isn't active."] HTTP status code: 400.

Your answer