Azure Machine Learning is not calling the current compute cluster but a deleted one.

Question

The Azure Machine Learning "job" fails because it cannot find a cluster I deleted. I have another cluster that I created and is "green" as in working, but I can't find where to change the backend.

Background.

I created the ML Environment with 50 resources using the "secure" Azure template. I deleted the clusters as the user that ran the bicep is not the same account as the end user. I created a CPU and GPU cluster- all seems fine.

I am following the "Azure-ML-In-a-Day" tutorial in the Azure ML Workspace.

The error is

Encountered error when building image: ScriptTask failed with: Unknown compute target 'cluster001'.

That was the old cluster name. I don't see where to change this in the sample code. I never entered the cluster in the sample code, so it must be somewhere. If I run Get-AzMLWorkspaceCompute, it lists the correct clusters. (mlw-cpu and mlw-gpu)

I even ran Remove-AzMLWorkspaceCompute -ResourceGroupName my-rg -WorkspaceName mlwcomputer -Name cluster001 -UnderlyingResourceAction 'Delete'

Any ideas?

Answer 1

I think the issue is was trying to use serverless and that may not work in the "secure" setup. This page

https://learn.microsoft.com/en-us/azure/machine-learning/how-to-secure-workspace-vnet?view=azureml-api-2&tabs=required%2Cpe%2Ccli#enable-azure-container-registry-acr

has

You can use the az ml workspace update command to set a build compute. The command is the same for both the v1 and v2 Azure CLI extensions for machine learning. In the following command, replace myworkspace with your workspace name, myresourcegroup with the resource group that contains the workspace, and mycomputecluster with the compute cluster name:

Azure CLICopy

az ml workspace update --name

You can switch back to serverless compute by executing the same command and referencing the compute as an empty space: --image-build-compute ''.

Answer 2

Hi Tom Sweet

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this!

Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others", I'll repost your solution.

Please click "Accept" the answer as original posters help the community find answers faster by identifying the correct answer.

Issue: Azure Machine Learning is not calling the current compute cluster but a deleted one. 

Resolution:I think the issue is was trying to use serverless and that may not work in the "secure" setup. This page

https://learn.microsoft.com/en-us/azure/machine-learning/how-to-secure-workspace-vnet?view=azureml-api-2&tabs=required%2Cpe%2Ccli#enable-azure-container-registry-acr

has

You can use the az ml workspace update command to set a build compute. The command is the same for both the v1 and v2 Azure CLI extensions for machine learning. In the following command, replace myworkspace with your workspace name, myresourcegroup with the resource group that contains the workspace, and mycomputecluster with the compute cluster name:

Azure CLICopy

YAMLCopy

az ml workspace update --name

You can switch back to serverless compute by executing the same command and referencing the compute as an empty space: --image-build-compute ''.

Please don’t forget to close the thread by clicking "Accept the answer" wherever the information provided helps you, as this can be beneficial to other community members.