Unable To Remove Azure AI Hub Via Azure Portal or CLI

Question

I have an Azure AI Hub I have been trying to delete (unsuccessfully) for several days. It is the only Hub that I have not been able to delete (despite the associated Storage Account no longer existing, either). When I look at the Events log in the Portal, I see the following:

When I run from an authenticated CLI:

az resource delete --ids /subscriptions/ff00e686-XXXXXXX

I get the following:

(InternalServerError) InternalServerError Code: InternalServerError Message: InternalServerError

This is frustrating because I am trying to clear my AI resources.

Answer 1

Hi there!

This is a common issue with Azure Machine Learning workspaces. The InternalServerError is a generic error, but the most frequent cause is lingering resources within the workspace, even if the associated storage account is gone. The Activity Log entries confirming the failures on Microsoft.MachineLearningServices/workspaces operations point directly to this.

Here's a comprehensive, step-by-step troubleshooting guide, prioritized by the most likely solutions:

1. Delete All Workspace Contents (Critical):

This is the most important step and often resolves the issue. You must meticulously remove all resources inside the workspace before attempting to delete the workspace itself.

• In the Azure Portal: Navigate to your Machine Learning workspace.
• Go through each blade in the left-hand navigation menu and delete the following:
  • Compute:
    • Compute Instances: Stop and delete all compute instances.
    • Compute Clusters: Delete all compute clusters.
    • Attached Computes: Delete all attached computes.
  • Endpoints: Delete all deployed endpoints (both real-time and batch endpoints). This is a very common cause of deletion problems.
  • Models: Delete all registered models.
  • Datasets: Delete all registered datasets.
  • Datastores: Delete any custom datastores (beyond the default one). If the associated storage account is already gone, this might be problematic, but try deleting all other resources first.
  • Experiments: Delete all experiments.
  • Pipelines: Delete all Machine Learning pipelines.
  • Environments: Delete all custom environments.
  • Notebooks: Delete all notebooks.
  • Connections: Delete all Connections.
• Be Thorough: Don't skip any of these sections. A single remaining resource can block deletion.

2. Check for Resource Locks:

• In the Azure Portal, navigate to your Machine Learning workspace.
• Go to the "Locks" blade.
• Remove any locks, especially "CanNotDelete" locks.

3. (Optional, but Recommended) Use Azure Resource Graph Explorer:

This helps identify any hidden or orphaned dependencies.

• Open the Azure Resource Graph Explorer in the Azure Portal.
• Run the following Kusto query (replace placeholders with your actual values):

 ```kusto
 resources
 | where type =~ 'microsoft.machinelearningservices/workspaces'
 | where name == 'your-workspace-name' // REPLACE with your workspace name
 | project id, name, type, properties, resourceGroup, subscriptionId

 //Then, check for resources related to this workspace
 resources
 | where properties contains '/subscriptions/ff00e686-XXXXXXX/resourcegroups/<YOUR RG>/providers/microsoft.machinelearningservices/workspaces/<YOUR WORKSPACE NAME>' //Replace with the resource id.
 | project id, name, type, properties, resourceGroup, subscriptionId
 ```

• Examine the results carefully for any related resources. Delete them if found.

4. Wait After Deletion:

• After deleting all the workspace contents (and any locks), wait for 15-30 minutes. This allows Azure to fully process the deletions.

5. Retry Deletion:

• Via Azure Portal: Go to the workspace's "Overview" blade and click "Delete."
• Via Azure CLI: Use the az resource delete command:

 ```bash
 az resource delete --ids /subscriptions/ff00e686-XXXXXXX.../resourceGroups/<YOUR_RG_NAME>/providers/Microsoft.MachineLearningServices/workspaces/<YOUR_WORKSPACE_NAME>
 ```
 (Replace the placeholders with your actual subscription ID, resource group name, and workspace name).
 *   **Via Azure PowerShell:**
    ```powershell
    $resourceId = "/subscriptions/ff00e686-XXXXXXX.../resourceGroups/<YOUR_RG_NAME>/providers/Microsoft.MachineLearningServices/workspaces/<YOUR_WORKSPACE_NAME>"
    Remove-AzResource -ResourceId $resourceId
    ```

6. Try Resource Group Deletion (If Applicable):

• If the Machine Learning workspace is the only remaining resource in its resource group, and you don't need the resource group itself, try deleting the entire resource group. This often works when individual resource deletion fails.

7. Contact Azure Support:

• If none of the above steps work, you must open a support ticket with Azure. Provide them with:
  • The full resource ID of the Machine Learning workspace.
  • The InternalServerError message and any other error details.
  • The steps you've already tried (including the cleanup of workspace contents).
  • Screenshots of the Activity Log (like the one you provided).
  • The timestamps of your deletion attempts.
• Azure support has backend tools that can force-delete resources in situations like this, but they need a support request to initiate that process.

Key Points:

• The most common solution is to meticulously delete all resources within the Machine Learning workspace before attempting to delete the workspace itself.
• Compute resources (instances, clusters) and deployed endpoints are frequent culprits.
• Use the Azure Resource Graph Explorer to help find hidden dependencies.
• Contact Azure Support if the problem persists after a thorough cleanup.

I hope this helps! Please let me know if you have any further questions or if the issue persists after trying these steps. Good luck!