Cannot Remove AI Hub Resource

Question

I am unable to delete my Azure AI Hub; it is the only Azure AI Hub associated with the Subscription that I have had trouble deleting. It appears to be in some odd state (it still lists itself through the Portal as associated with a Storage Account that no longer exists). I cannot delete as the Account Owner through Azure Portal or CLI.

Here is the Activity log for the resource demonstrating multiple remove attempts:

When running the az resource delete --ids command through the CLI on the resource, I get:

(InternalServerError) InternalServerError Code: InternalServerError Message: InternalServerError

Answer 1

Hi there!

This is a common issue with Azure Machine Learning workspaces. The InternalServerError is a generic error, but the most frequent cause is lingering resources within the workspace, even if the associated storage account is gone. The Activity Log entries confirming the failures on Microsoft.MachineLearningServices/workspaces operations point directly to this.

Here's a comprehensive, step-by-step troubleshooting guide, prioritized by the most likely solutions:

1. Delete All Workspace Contents (Critical):

This is the most important step and often resolves the issue. You must meticulously remove all resources inside the workspace before attempting to delete the workspace itself.

• In the Azure Portal: Navigate to your Machine Learning workspace.
• Go through each blade in the left-hand navigation menu and delete the following:
  • Compute:
    • Compute Instances: Stop and delete all compute instances.
    • Compute Clusters: Delete all compute clusters.
    • Attached Computes: Delete all attached computes.
  • Endpoints: Delete all deployed endpoints (both real-time and batch endpoints). This is a very common cause of deletion problems.
  • Models: Delete all registered models.
  • Datasets: Delete all registered datasets.
  • Datastores: Delete any custom datastores (beyond the default one). If the associated storage account is already gone, this might be problematic, but try deleting all other resources first.
  • Experiments: Delete all experiments.
  • Pipelines: Delete all Machine Learning pipelines.
  • Environments: Delete all custom environments.
  • Notebooks: Delete all notebooks.
  • Connections: Delete all Connections.
• Be Thorough: Don't skip any of these sections. A single remaining resource can block deletion.

2. Check for Resource Locks:

• In the Azure Portal, navigate to your Machine Learning workspace.
• Go to the "Locks" blade.
• Remove any locks, especially "CanNotDelete" locks.

3. (Optional, but Recommended) Use Azure Resource Graph Explorer:

This helps identify any hidden or orphaned dependencies.

• Open the Azure Resource Graph Explorer in the Azure Portal.
• Run the following Kusto query (replace placeholders with your actual values):

 resources
 | where type =~ 'microsoft.machinelearningservices/workspaces'
 | where name == 'your-workspace-name' // REPLACE with your workspace name
 | project id, name, type, properties, resourceGroup, subscriptionId

 //Then, check for resources related to this workspace
 resources
 | where properties contains '/subscriptions/ff00e686-XXXXXXX/resourcegroups/<YOUR RG>/providers/microsoft.machinelearningservices/workspaces/<YOUR WORKSPACE NAME>' //Replace with the resource id.
 | project id, name, type, properties, resourceGroup, subscriptionId

• Examine the results carefully for any related resources. Delete them if found.

4. Wait After Deletion:

• After deleting all the workspace contents (and any locks), wait for 15-30 minutes. This allows Azure to fully process the deletions.

5. Retry Deletion:

• Via Azure Portal: Go to the workspace's "Overview" blade and click "Delete."
• Via Azure CLI: Use the az resource delete command:

 az resource delete --ids /subscriptions/ff00e686-XXXXXXX.../resourceGroups/<YOUR_RG_NAME>/providers/Microsoft.MachineLearningServices/workspaces/<YOUR_WORKSPACE_NAME>

(Replace the placeholders with your actual subscription ID, resource group name, and workspace name).

• Via Azure PowerShell:

$resourceId = "/subscriptions/ff00e686-XXXXXXX.../resourceGroups/<YOUR_RG_NAME>/providers/Microsoft.MachineLearningServices/workspaces/<YOUR_WORKSPACE_NAME>"
Remove-AzResource -ResourceId $resourceId

6. Try Resource Group Deletion (If Applicable):

• If the Machine Learning workspace is the only remaining resource in its resource group, and you don't need the resource group itself, try deleting the entire resource group. This often works when individual resource deletion fails.

7. Contact Azure Support:

• If none of the above steps work, you must open a support ticket with Azure. Provide them with:
  • The full resource ID of the Machine Learning workspace.
  • The InternalServerError message and any other error details.
  • The steps you've already tried (including the cleanup of workspace contents).
  • Screenshots of the Activity Log (like the one you provided).
  • The timestamps of your deletion attempts.
• Azure support has backend tools that can force-delete resources in situations like this, but they need a support request to initiate that process.

Key Points:

• The most common solution is to meticulously delete all resources within the Machine Learning workspace before attempting to delete the workspace itself.
• Compute resources (instances, clusters) and deployed endpoints are frequent culprits.
• Use the Azure Resource Graph Explorer to help find hidden dependencies.
• Contact Azure Support if the problem persists after a thorough cleanup.

I hope this helps! Please let me know if you have any further questions or if the issue persists after trying these steps. Good luck!

Answer 2

Hi @Thomas Eaves,

I'm glad to hear that your issue has been resolved. And thanks for sharing the information, which might be beneficial to other community members reading this thread as solution. Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", so I'll reiterate your response to an answer in case you'd like to accept the answer. This will help other users who may have a similar query find the solution more easily.

Query: Cannot Remove AI Hub Resource

Response: It ended up being an issue with a Blade (Azure AI Hub dependency) not having been purged.

If you have any further questions or concerns, please don't hesitate to ask. We're always here to help.

---

Do click Accept Answer and Yes for was this answer helpful.