Azure Synapse Notebook does't fail immediately in case of error in code

Question

Azure Synapse Notebook does't fail immediately in case of error in code

Peter Polacek 25

Hello,

I am struggling with really weird behavior when executing a notebook within Azure Synapse. I am utilizing pyspark and sql within this notebook and noticed that the notebook doesn't fail immediately in case of a bug or let's say typo in a code. For example I create a temp view and if I then make a SELECT from that view in another cell and type wrong view name (view which doesn't exist) the cell runs anyway very long time (around 30 minutes) and then it fails on some generic error:

"org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception:"

I'd expect that it fails immediately on "object doesn't exist" or whatever.

I encountered the same behavior also when using python code which makes all the debugging and testing very hard.

Do you please know why it's like that and how to eventually avoid that ?

Thanks

Chandra Boorla 10,410 Reputation points Microsoft External Staff

2025-01-23T20:48:32.16+00:00

Hi @Peter Polacek

Greetings & Welcome to Microsoft Q&A forum! Thanks for posting your query!

The behavior you're experiencing in Azure Synapse Notebooks, where errors do not cause immediate failures and instead result in long execution times before failing, is not uncommon in distributed computing environments like Spark. This can occur due to the way Spark processes jobs. When you submit a job, it may take time to schedule and execute, especially if it involves querying metadata or executing tasks across a cluster.

In your case, when you create a temporary view and then attempt to query it with a non-existent name, Spark may not immediately check for the existence of the view until it tries to execute the query, which can take a significant amount of time, especially if the query involves complex operations or large datasets.

To mitigate this issue, you can implement the following strategies:

Check for View Existence - Before executing a query that references a temporary view, you can check if the view exists programmatically. This can be done by querying the metadata or using a try-except block to catch exceptions early.

Use Smaller Queries - Break down your queries into smaller parts. This way, if there's an error in a specific part, it will fail faster, allowing you to debug more efficiently.

Error Handling - Implement error handling in your code to catch exceptions and provide more informative messages, which can help in debugging.

Update Spark and Hive Versions - If you're using an older Spark pool, consider upgrading to the latest version in Synapse, as newer versions may have improved error reporting and reduced retries.

For more details, please refer: https://learn.microsoft.com/en-us/azure/hdinsight/share-hive-metastore-with-synapse

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Thank you.
Chandra Boorla 10,410 Reputation points Microsoft External Staff

2025-01-24T19:21:41.7766667+00:00

@Peter Polacek

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Peter Polacek 25 Reputation points

2025-01-24T19:51:43.0433333+00:00

Hello @Chandra Boorla

thank you for you response.

Unfortunately I didn't find any satisfying solution so far.

I am posting another example where the execution of a cell took more than 20 minutes until it failed even I would expect that it would fail immediately on such kind of error.
Smaran Thoomu 21,600 Reputation points Microsoft External Staff

2025-01-27T11:06:49.0133333+00:00

@Peter Polacek We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Chandra Boorla 10,410 Reputation points Microsoft External Staff

2025-01-28T10:12:15.11+00:00

@Peter Polacek

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

1 answer

Your answer

Chandra Boorla 10,410 Reputation points Microsoft External Staff

2025-01-24T19:21:41.7766667+00:00

@Peter Polacek

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Peter Polacek 25 Reputation points

2025-01-24T19:51:43.0433333+00:00

Hello @Chandra Boorla

thank you for you response.

Unfortunately I didn't find any satisfying solution so far.

I am posting another example where the execution of a cell took more than 20 minutes until it failed even I would expect that it would fail immediately on such kind of error.
Smaran Thoomu 21,600 Reputation points Microsoft External Staff

2025-01-27T11:06:49.0133333+00:00

@Peter Polacek We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Chandra Boorla 10,410 Reputation points Microsoft External Staff

2025-01-28T10:12:15.11+00:00

@Peter Polacek

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Answer 1

Chandra Boorla 10,410 Microsoft External Staff

@Peter Polacek

Apologies for the inconvenience.

The error message you're encountering indicates that there is an issue with accessing Azure Blob Storage (specifically, the Azure Data Lake Storage Gen2) from your Hive query. The InvalidAbfsRestOperationException suggests that there was a problem with the REST operation being performed on the Azure Blob File System. This can happen due to several reasons such as permissions issues, incorrect configurations, or even network problems.

Please refer to the following similar thread links on Q&A platform to get some useful insights.

https://learn.microsoft.com/en-us/answers/questions/565482/i-am-getting-error-while-running-the-spark-job-wit

https://learn.microsoft.com/en-us/answers/questions/1605142/data-flow-run-fails-with-invalidabfsrestoperatione

For additional information, please refer: https://azureplayer.net/2022/12/invalidabfsrestoperationexception-in-azure-synapse-notebook/

Disclaimer: This response contains a reference to a third-party World Wide Web site. Microsoft is providing this information as a convenience to you. Microsoft does not control these sites and has not tested any software or information found on these sites; therefore, Microsoft cannot make any representations regarding the quality, safety, or suitability of any software or information found there. There are inherent dangers in the use of any software found on the Internet, and Microsoft cautions you to make sure that you completely understand the risk before retrieving any software from the Internet.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Thank you.

Peter Polacek 25 Reputation points

2025-01-29T08:20:18.8133333+00:00

Hello,

thank you for response, however I know where is the problem in my code and that was just an example. The problem is that if I don't know and I need to investigate, that is quite problematic because I need to wait long time until it fails. I mean, I don't know if there's an issue in my code or the processing just takes long.

As another example, I was currently working with spark sql, I created temp views and then joined them to get the desired results. I used the same query I used before in serverless SQL, but it didn't work because some function were not compatible (for example DATEADD in serverless SQL pool and DATE_ADD in spark SQL). I didn't know that before and as the query didn't fail immediately, I assumed it's correct in terms of syntax, etc and I thought it just need to process long. I would expect that it would fail immediately in such case and it just makes all the code debugging quite difficult.
Chandra Boorla 10,410 Reputation points Microsoft External Staff

2025-01-29T19:09:38.9033333+00:00
@Peter Polacek

Apologies for the inconvenience that you are facing here.

The behavior you are experiencing, where errors do not fail immediately in Azure Synapse Notebooks, is a result of how Spark handles execution and job orchestration. Spark uses a lazy evaluation model, meaning transformations (like creating a temporary view or defining a query) are not executed immediately. Instead, the execution happens when an action (such as show, collect, or a SQL query execution) is triggered. This design allows Spark to optimize the execution plan, but it also means errors may not surface until the job is executed, which can be frustrating for debugging.

Why Errors Take Time to Surface

When a query is executed, Spark needs to allocate resources across the cluster. If there’s a bottleneck or a large cluster setup, this can add significant delay.

Spark retries tasks multiple times by default in case of failures (e.g., missing data, connection issues). This can delay failure detection.

Operations involving metadata (e.g., temporary views) or external systems (e.g., Azure Blob Storage) might hang while waiting for a response, only failing after a timeout.

Appreciate if you could share the feedback on our feedback channel. Which would be open for the user community to upvote & comment on. This allows our product teams to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature.

I hope this information helps.

Thank you.
Peter Polacek 25 Reputation points

2025-01-30T15:13:48.2666667+00:00

Thank you for explanations and suggestion @Chandra Boorla . I posted new idea in that feedback channel.

Thanks!
Chandra Boorla 10,410 Reputation points Microsoft External Staff

2025-01-30T16:42:06.5633333+00:00

@Peter Polacek

Following up to see if the above suggestion was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Share via

Azure Synapse Notebook does't fail immediately in case of error in code

1 answer

Your answer