How to use linked service in Notebook with pyspark

Question

I have pyspark script in Notebook to read and write data in ADLS Gen2. Below is an sample of the pyspark script. But in the Synapse I only have a linked service created with Service Principle could connect to the ADLS Gen2, so I need to specify in notebook to use that linked service to make the connection. But how could I do this in notebook?

df.write.mode("overwrite").parquet("abfss://******@xxxxxxxxxxxx.dfs.core.windows.net/test")

Answer 1

Hello Shao Peng Sun,

Welcome to the MS Q&A platform.

You can use the storage_options parameter in your PySpark script to specify the linked service for reading and writing data in ADLS Gen2. You can modify your script to below to use the linked service:

linked_service_name = 'your_linked_service_name'

# read data

df = spark.read.parquet(
    "abfss://******@bhargavasynapsegen2.dfs.core.windows.net/NYCTrip.parquet",
    storage_options={'linked_service': linked_service_name}
)
df.show()

# write
spark.conf.set("fs.azure.account.auth.type.<your-storage-account-name>.dfs.core.windows.net", "OAuth")
spark.conf.set("fs.azure.account.oauth.provider.type.<your-storage-account-name>.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")
spark.conf.set("fs.azure.account.oauth2.client.id.<your-storage-account-name>.dfs.core.windows.net", "<your-service-principal-client-id>")
spark.conf.set("fs.azure.account.oauth2.client.secret.<your-storage-account-name>.dfs.core.windows.net", "<your-service-principal-client-secret>")
spark.conf.set("fs.azure.account.oauth2.client.endpoint.<your-storage-account-name>.dfs.core.windows.net", "https://login.microsoftonline.com/<your-tenant-id>/oauth2/token")


df.write.mode("overwrite").parquet("abfss://******@xxxxxxxxxxxx.dfs.core.windows.net/test")

Please see the below screenshot for your reference.

Using this, I was able to read my parquet file.

I hope this help.

If this answers your question, please consider accepting the answer by hitting the Accept answer and up-vote as it helps the community look for answers to similar questions.

Answer 2

So we still have to use the storage account name. Which means, from the point of view of being able to deploy environment-aware linked services and use them from environment-unaware code, it's pointless.