Init scripts stored in volumes are not getting executed via Job Shared cluster via ADF

Question

Hello Team,

We have stored our init script in Azure databricks managed volumes path and we want to execute notebook via Job Shared cluster, the pipeline we are triggering is via ADF but the job cluster gives an error as

We are using Assigned Manged Identity as an authentication from adf to databricks and this identity has all the permissions on databricks resource and clusters, objects, also the adf service principal has all permissions.

Please let us know why is this issue occuring and how to resolve it. Since as per Microsoft article Shared Compute cluster, volumes are supported.

Regards,

Ashwini G

Answer 1

Hi @Ashwini Gaikwad
Welcome to Microsoft Q&A platform and thanks for posting your query here.

Based on our internal review, here are some suggestions to help resolve the issue:

1. Log Examination: Please check the logs of the Job Shared cluster to identify any specific error messages related to the init script execution. This could offer more insights into the root cause of the issue.
2. Allowlist Configuration for Shared Access Mode: For clusters operating in shared access mode, it is essential to ensure that init scripts are added to the allowlist. Without this, the scripts may not execute correctly. For guidance on how to configure the allowlist for libraries and init scripts on shared compute clusters, you can refer to this documentation.

You can do something like below:

I hope this information helps. If you have any further questions, please let me know.