Read data from Databricks Delta Sharing to Azure Synapse

Question

Hi,

I am trying to read the data from Databricks Delta Sharing and write the data to Azure Synapse by referring this documentation https://docs.databricks.com/en/connect/external-systems/synapse-analytics.html#language-python

I have constructed below Python scripts:

        data_frame.write \
            .format("com.databricks.spark.sqldw") \
            .option("url", config['synapse']['connection_string']) \
            .option("forwardSparkAzureStorageCredentials", "true") \
            .option("dbTable", f"{schema_name}.{table.name}") \
            .option("tempDir", temp_dir) \
            .save()

But it shows below error:

"

org.apache.spark.SparkClassNotFoundException: [DATA_SOURCE_NOT_FOUND] Failed to find the data source: com.databricks.spark.sqldw. Please find packages at https://spark.apache.org/third-party-projects.html.

"

How can I resolve this error?

Thanks.

Answer 1

Hello @Cheng, Jack

Thanks for the question and using MS Q&A platform.

The error you are encountering, org.apache.spark.SparkClassNotFoundException: [DATA_SOURCE_NOT_FOUND] Failed to find the data source: com.databricks.spark.sqldw, indicates that the Spark SQL Data Warehouse (SQL DW) connector is not available in your current Spark environment. This can happen if the required library is not included in your Databricks cluster.

To resolve this error, you should ensure that the appropriate library for the SQL DW connector is installed on your Databricks cluster. You can do this by following these steps:

1. Go to your Databricks workspace.
2. Navigate to the "Clusters" section.
3. Select your cluster and click on the "Libraries" tab.
4. Install the com.databricks:spark-sqldw_2.11:<version> library, replacing <version> with the specific version compatible with your Spark version.

Once the library is installed, try running your script again.

Similar thread for reference: Error Connecting to Delta Share

Hope this helps. Do let us know if you have any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.