Need to have more clearance spark synapse copy write options
Girisankar Rajendran
0
Reputation points
Hi All,
For my business requirements, my spark code in databricks were used to load a gigabytes of data into Azure synapse database.
The actual issue was by using COPY Into I can able to use option errorfile used to store error records.
But in spark I am unable to achieve it. Could any one help us to achieve this.
Spark code:
spark.conf.set("fs.azure.account.auth.type", "OAuth")
spark.conf.set("fs.azure.account.oauth.provider.type", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")
spark.conf.set("fs.azure.account.oauth2.client.id", "
Sign in to answer