Is Azure Synapse Analytics expected to support Spark 3.5 in the near future?

Question

Is Azure Synapse Analytics expected to support Spark 3.5 in the near future?

Spark 3.5 allows to use Delta Lake 3.0 version which improved overall performance significantly. https://delta.io/blog/delta-lake-performance/

Apache Spark 3.5 compatible with Delta Lake 3.0 and higher version https://docs.delta.io/latest/releases.html

current version link https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-version-support

Answer 1

Hi @Akjol Syeryekkhaan
Welcome to Microsoft Q&A platform and thanks for posting your query here.

As of now, Azure Synapse Analytics supports Apache Spark up to version 3.4, which became generally available on April 8, 2024.

The support for Apache Spark 3.5 is currently under evaluation, and there is no confirmed timeline for its release within Synapse. Given that support for Spark 3.3 is scheduled to end on March 31, 2025, it is recommended to migrate your workloads to Spark 3.4 to ensure continued support and to take advantage of the latest improvements.

Regarding Delta Lake, Spark 3.4 in Synapse includes Delta Lake version 2.2.0. While Delta Lake 3.0 offers significant performance enhancements, it requires Spark 3.5 or higher, which is not yet available in Synapse. Therefore, to utilize Delta Lake 3.0's features, support for Spark 3.5 in Synapse would be necessary.

In the meantime, if you're looking to leverage the performance improvements offered by Delta Lake 3.0, you might consider exploring Microsoft Fabric. Microsoft Fabric's Runtime 1.3 is based on Apache Spark 3.5, providing compatibility with Delta Lake 3.0 and its associated enhancements.

For the most current information on runtime support and updates, please refer Azure Synapse documentation.

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.