Need to have more clearance spark synapse copy write options

Girisankar Rajendran 0 Reputation points
2024-03-22T07:34:45.9733333+00:00

Hi All,

For my business requirements, my spark code in databricks were used to load a gigabytes of data into Azure synapse database.

The actual issue was by using COPY Into I can able to use option errorfile used to store error records.

But in spark I am unable to achieve it. Could any one help us to achieve this.

Spark code:

spark.conf.set("fs.azure.account.auth.type", "OAuth")

spark.conf.set("fs.azure.account.oauth.provider.type", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")

spark.conf.set("fs.azure.account.oauth2.client.id", "

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,093 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,285 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,049 questions
Azure Dedicated Host
Azure Dedicated Host
An Azure service that provides a dedicated physical server to host Azure virtual machines for Windows and Linux.
51 questions
{count} votes

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.