Azure Databricks - Database Error

Question

I created the following database in Azure Databricks -

CREATE DATABASE IF NOT EXISTS f1_processed

LOCATION "/mnt/<Container_Name>/<Folder_Name>"

Database got created under "hive_metastore"

Then I'm trying to create a table (with data's from a data frame) in the above database with the following command -

<DataFrame_Name>.write.mode("overwrite").format("parquet").saveAsTable(name="f1_processed.results", partitionBy="id")

It throws the below error -

The schema databrickscourse_ws.f1_processed cannot be found.

Why my table creation command is trying to find Database(Schema) (f1_processed) in databrickscourse_ws?

Can anyone please help ?

Answer 1

Hi @Ramanathan Murugan Veerabagu
Welcome to Microsoft Q&A platform and thanks for posting your query here.

It seems like you are encountering an error while trying to create a table in the "f1_processed" database in Azure Databricks. The error message indicates that the schema "databrickscourse_ws.f1_processed" cannot be found, which suggests that there might be an issue with the database reference in your table creation command.

When you create a database in Azure Databricks, it is typically associated with a specific workspace or context. In your case, it seems that the database "f1_processed" is being referenced within the context of the "databrickscourse_ws" schema, which is causing the error.

To resolve this issue, you can explicitly specify the database name when creating the table to ensure that it is created in the correct database. Here is an updated version of your table creation command with the database name specified:

<DataFrame_Name>.write.mode("overwrite").format("parquet").saveAsTable(name="f1_processed.results", database="f1_processed", partitionBy="id")

By explicitly providing the database name in the table creation command, you can ensure that the table is created in the "f1_processed" database without referencing the incorrect schema "databrickscourse_ws".

If you continue to encounter any issues or need further assistance, please feel free to provide more details or ask for additional help!
Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.