How to auto-authenticate AML compute instance to start on a regular schedule?

Caiwen Wu 0 Microsoft Employee

Hi, I am trying to use Azure ML workspace to run Python notebooks on a regular basis to query Kusto data and conduct data processing. I created a Compute instance in AML, set up the starter shell script, and configured the Startup schedule of the compute instance. I also used User assigned identity as the authentication method. However, the Compute instance does not always start successfully with the schedule, when it failed, a banner would appear: "Your compute needs to be authenticated before you can mount data to it. Please use the authenticate button to sign in to receive the authentication token." How can I resolve it? Thank you!

Vikram Singh 980 Reputation points Microsoft Employee

2025-01-17T06:51:43.1166667+00:00
Hello @Caiwen Wu ,

Thanks for posting your question in Microsoft Q&A, apologize for any inconvenience caused on this.

To resolve authentication issues with Azure ML compute instances during scheduled startups, please ensure that your User Assigned Identity has the necessary permissions. Additionally, verify that your startup script includes the correct authentication code. To assist in troubleshooting, kindly provide the authentication code snippet and relevant logs.

In the meantime, you can update your authentication method using the following example:

from azure.identity import ManagedIdentityCredential from azure.kusto.data import KustoClient, KustoConnectionStringBuilder # Authenticate using User Assigned Identity credential = ManagedIdentityCredential(client_id='YOUR_USER_ASSIGNED_IDENTITY_CLIENT_ID') # Set up Kusto connection kusto_cluster = "https://<your-kusto-cluster>.kusto.windows.net" kusto_database = "<your-database>" kcsb = KustoConnectionStringBuilder.with_aad_managed_identity(kusto_cluster, credential) client = KustoClient(kcsb) # Now you can run your queries

This code sample demonstrates how to authenticate using a User Assigned Identity and set up a connection to an Azure Data Explorer (Kusto) cluster. Please update the client_id, kusto_cluster, and kusto_database with your specific details.

Also let me know if it helped!

Thank you
Caiwen Wu 0 Reputation points Microsoft Employee

2025-01-18T03:20:55.28+00:00
Hi @vikram singh , thanks for your help! I tried the authentication code above but hit an error.

Here is what I used in my python script. It works fine if I manually start the Compute instance to run the notebook, however, for scheduled startups, it will fail in the middle of the notebook and the banner would pop up on the Compute page: "Your compute needs to be authenticated before you can mount data to it. Please use the authenticate button to sign in to receive the authentication token." Any thoughts on how to resolve that? Thank you very much!

from azure.kusto.data import KustoClient, KustoConnectionStringBuilder # Set up Kusto connection kusto_cluster = "https://exptelemetrysds.westus2.kusto.windows.net/" client_id = "<YOUR_USER_ASSIGNED_IDENTITY_CLIENT_ID>" kusto_database = "FRED" KustoConnectionStringBuilder.with_aad_managed_service_identity_authentication(kusto_cluster, client_id) client = KustoClient(kcsb)
Caiwen Wu 0 Reputation points Microsoft Employee

2025-01-18T03:21:51.2066667+00:00

BTW when I search online, I can't find this syntax for Python: kcsb = KustoConnectionStringBuilder.with_aad_managed_identity(kusto_cluster, credential).
Vikram Singh 980 Reputation points Microsoft Employee

2025-01-20T06:24:43.9866667+00:00
Hello @Caiwen Wu ,

Thank you for your detailed description of the issue. Here are some suggestions to help resolve it:

Ensure Authentication: Confirm that your startup script includes the necessary authentication logic at the start to complete the authentication before any operations:

from azure.identity import ManagedIdentityCredential from azure.kusto.data import KustoClient, KustoConnectionStringBuilder # Authenticate using User Assigned Identity credential = ManagedIdentityCredential(client_id='YOUR_USER_ASSIGNED_IDENTITY_CLIENT_ID') # Set up Kusto connection using Managed Identity kusto_cluster = "https://exptelemetrysds.westus2.kusto.windows.net/" kcsb = KustoConnectionStringBuilder.with_aad_managed_identity_authentication(kusto_cluster, client_id='YOUR_USER_ASSIGNED_IDENTITY_CLIENT_ID') client = KustoClient(kcsb)

Update SDK: Ensure you're using the latest version of the Azure Kusto SDK:

pip install azure-kusto-data --upgrade

Refer to GitHub Examples: For detailed implementation guidance, refer to:

Azure Kusto Python SDK - Sample Usage

Azure Kusto Python SDK - README

Temporary Workaround: Add a manual authentication step (az login) in your startup script as a temporary fix to test if the issue is related to identity authentication during scheduled starts. However, this is not recommended for long-term use.

Note: Issues might arise from token handling or authentication timing during scheduled runs. Ensure your script handles token acquisition and refresh properly to avoid these issues.

Let me know if this helps or if you need further assistance! Feel free to reach out for more detailed guidance through a private channel.
Vikram Singh 980 Reputation points Microsoft Employee

2025-01-21T04:08:11.2433333+00:00

Hello @Caiwen Wu ,

Greetings.

Just following up to check if my suggestion helped. Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

Thank you
Caiwen Wu 0 Reputation points Microsoft Employee

2025-01-21T21:28:55.6666667+00:00

Hi Vikram Singh, thanks for your follow-up response! I have included the authentication script at the top of my startup script, however, the program still stopped in the middle of it. The compute instance started at scheduled time but stopped before finishing the program, with the same banner showing:" Your compute needs to be authenticated before you can mount data to it. Please use the authenticate button to sign in to receive the authentication token." So I suspect it to be related to the authentication token somehow? Could you please provide your alias so I can reach out offline via email/chat? Thanks very much in advance!

1 answer

Vikram Singh 980 Reputation points Microsoft Employee

2025-02-05T05:03:16.9+00:00
Hello @Caiwen Wu ,

As discussed, you can run Code as AML Jobs Instead of Notebooks. To automate scripts as jobs/pipelines (instead of interactive notebooks), please find the below docs:

AML Jobs Overview: Create and submit jobs: Covers CLI v2 and SDK options for running code as jobs. Microsoft Docs: Submit and track Azure ML jobs

Convert Notebooks to Jobs: Run notebooks as jobs: Steps to submit a notebook directly as an AML job. Microsoft Docs: Run Jupyter Notebooks as jobs

AML Pipelines: Design workflows to automate multi-step tasks. Create pipelines for recurring jobs: Microsoft Docs: Create ML pipelines.

Schedule Jobs: Use AML Pipelines with schedules or Azure Logic Apps. Trigger jobs on a schedule:

Why Use Jobs Instead of Notebooks?

Jobs are designed for unattended execution (no interactive authentication required).

Jobs leverage the workspace’s managed identity by default.

Jobs can run on dedicated compute clusters, which auto-scale and cost less than persistent compute instances.

Logs and outputs are stored automatically in the AML workspace.

Running Code as Azure Machine Learning (AML) Jobs Instead of Notebooks:

Submitting Training Jobs: Azure Machine Learning provides multiple methods to submit training jobs, allowing you to run your code as a job rather than within a notebook environment. This approach is beneficial for scaling and automating your machine learning workflows. For detailed instructions on configuring and submitting training jobs, consult the following resource: learn.microsoft.com

Running a Training Script as a Command Job: To convert your existing code into a script and execute it as a command job in Azure Machine Learning, you can follow the guidelines provided in this module: learn.microsoft.com

If the reply was helpful please don't forget to upvote and/or accept as answer,

Thank you
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

How to auto-authenticate AML compute instance to start on a regular schedule?

1 answer

Your answer