REST API : Service Principal gets 403 error when reading tables (spark.sql) on Fabric lakehouse (schema enabled)

Question

There are many similar posts, however, they did not help me get it to work.
I have a schema-enabled lakehouse and use a service principal to execute a notebook that reads/writes data into the lakehouse.
When I read/write into a lakehouse without schema it works fine.

from pyspark.sql import SparkSession

# Initialize Spark session
spark = SparkSession.builder \
    .appName("SaveAsTableExample") \
    .getOrCreate()
    
# Create a list of tuples
data = [("Alice", 34), ("Bob", 45), ("Cathy", 29)]

# Define the schema (column names)
columns = ["Name", "Age"]

# Create the DataFrame
df = spark.createDataFrame(data, columns)
# # Save DataFrame as a table
df.write.mode("overwrite").saveAsTable("people2")
df.write.format("delta").saveAsTable("test2")

Here are the official docs that I used
https://learn.microsoft.com/en-us/fabric/data-engineering/notebook-public-api

• I manually attached the lakehouse
• I sent the lakehouse info in the configuration section
• I also tried following.

# %%pyspark 
# !echo "spark.trident.pbiApiVersion=v1">>/home/trusted-service-user/.trident-context

Answer 1

Hi Okan Alaftekin,

Thank you for reaching out to us on the Microsoft Q&A forum.

Currently, Microsoft Fabric is not supported on the Q&A forums. You can find the list of supported products here.

We recommend posting your query in the dedicated Microsoft Fabric Community Forum under the General Discussion section. Experts and moderators are available there to provide assistance and guidance. You can access the community here.

I hope this information is helpful. If it is, please click Accept Answer and Upvote to assist other community members.