Import databricks data using tabular editor

Question

I imported data from databricks to AAS using the Simba ODBC driver but after deploying the model, i can't seem to make the refresh work.
I get this error "Failed to save modifications to the server. Error returned: 'Expression in partition '<oii>my_partion_name</oii>' in table '<oii>my_table_name</oii>' references unknown variable or function 'Databricks.Catalogs'"

Answer 1

Hi George Matlhogela,

Thanks for reaching out to Microsoft Q&A.

The error indicates that the partition is referencing a function or object (Databricks.Catalogs) that AAS could not recognize.

Here are a few steps you can follow to troubleshoot and resolve the issue:

1. Check ODBC Connection:

Ensure that the Simba ODBC driver is properly configured, and the connection string used in the model is correct. Verify that the connection works as expected outside of AAS (ex., in PBI or SSMS).

2. Review M Query:

The error suggests an issue with the M query or data source expression. If you are using Tabular Editor or any custom M script, review the query inside the partition and make sure that it doesn't reference unsupported functions or objects. AAS might not fully support all functions available in Databricks, so ensure the query only uses supported constructs.

3. Use Power Query for Databricks:

Consider using Power Query to connect to Databricks, as it might handle the connection and refresh better within AAS.

You can do this by:

• Opening your model in Tabular Editor.
• Modifying the data source connection to use Power Query to pull data from Databricks.

4. Test Direct Query:

If you're using an Import mode, consider switching to DirectQuery mode in Azure Analysis Services and test if the data refreshes properly. Databricks typically performs well with DirectQuery due to the distributed nature of its data processing.

5. Check Data Source Permissions:

Ensure that the AAS service principal or user has proper permissions to access the Databricks data. Sometimes, permission issues can cause refresh failures.

6. ODBC Driver Compatibility:

Ensure that the Simba ODBC driver is the latest version and that it is fully compatible with AAS. You might need to update the driver or verify that all necessary drivers are installed on the AAS server.

7. Deploy with Partition Expressions:

If you are deploying the model using Tabular Editor, ensure that the partition expressions are correctly specified and valid. Incorrect expressions can cause issues when the model tries to process or refresh data.

8. Logs and Diagnostics:

Check the AAS logs for more detailed error messages, which might provide more insights into why the Databricks.Catalogs reference is failing. You can access AAS diagnostics logs in Azure portal under the "Diagnostics Settings" of the AAS instance.

If these steps do not resolve the issue, you might want to consider reevaluating the query approach or using a different method for importing data from Databricks into AAS.

Please 'Upvote'(Thumbs-up) and 'Accept' as an answer if the reply was helpful. This will benefit other community members who face the same issue.