Hello everyone, I am facing an issue while deploying an Azure Machine Learning setup with private endpoints. Here’s what I have done so far: Provisioned an Azure Storage account with private endpoints for both Blob and File services. Deployed an Azure Machine Learning workspace, Container Registry, Application Insights, and Key Vault. Configured all resources in a shared virtual network with private endpoints enabled. The Issue: While running a training pipeline stage that registers a dataset, I encounter the following error: Error in creating data asset with Exception:

Hello Gupta, Shalu , Welcome to the Microsoft Q&A and thank you for posting your questions here. I understand that you are having issue while deploying an Azure Machine Learning setup with private endpoints. Most of all, the error you got, "Failed to establish a new connection: [Errno -2] Name or service not known," typically points to DNS resolution issues. However, the lists of a few steps below and links associated provide more steps and reading for better understand of related error on how you can resolve the issue. Your DNS settings should be correctly configured, if you're using Azure DNS, verify that the private DNS zones for privatelink.blob.core.windows.net and privatelink.file.core.windows.net are correctly linked to your virtual network - and if you're using a custom DNS solution, make sure it can resolve the necessary FQDNs. You can test this by running nslookup <fqdn> from a VM within the same virtual network. - https://learn.microsoft.com/en-us/azure/machine-learning/how-to-troubleshoot-secure-connection-workspace?view=azureml-api-2 Double-check your private endpoints is associated with the appropriate resources and recreating the private endpoints can resolve configuration issues. - https://learn.microsoft.com/en-us/azure/machine-learning/how-to-configure-private-link?view=azureml-api-2 Verify that your NSGs allow traffic to and from the necessary services.- https://learn.microsoft.com/en-us/azure/machine-learning/how-to-configure-private-link?view=azureml-api-2 Should there be any proxy, it might be interfering with the connection. Try disabling the proxy temporarily to see if it resolves the issue. - https://learn.microsoft.com/en-us/azure/machine-learning/how-to-troubleshoot-secure-connection-workspace?view=azureml-api-2 Use the Azure Machine Learning workspace diagnostic API to identify any configuration problems with your workspace. - https://learn.microsoft.com/en-us/azure/machine-learning/how-to-troubleshoot-secure-connection-workspace?view=azureml-api-2 You've mentioned that the roles are configured, but it's worth double-checking that there are no missing permissions. - https://learn.microsoft.com/en-us/azure/machine-learning/how-to-configure-private-link?view=azureml-api-2 . The above are areas the which the error might be troubleshoot. Except there is general (temporary) downtime and/or IP Address issue. I hope this is helpful! Do not hesitate to let me know if you have any other questions. Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.

Azure Machine Learning Endpoint Deployment

Gupta, Shalu 30

Hello everyone,

I am facing an issue while deploying an Azure Machine Learning setup with private endpoints.

Here’s what I have done so far:

Provisioned an Azure Storage account with private endpoints for both Blob and File services.
Deployed an Azure Machine Learning workspace, Container Registry, Application Insights, and Key Vault.
Configured all resources in a shared virtual network with private endpoints enabled.

The Issue: While running a training pipeline stage that registers a dataset, I encounter the following error:

Error in creating data asset with Exception:

romungi-MSFT 47,206 Reputation points Microsoft Employee

2024-11-22T13:06:05.7566667+00:00
@Gupta, Shalu A setup like this needs a lot of checkboxes to be checked since it is a custom setup with a secure workspace. To start debugging from the error, it looks like a certain connection or route is failing. Usually, an error to establish a connection provides the connection or service name. Something like below:

(Caused by NewConnectionError(': Failed to establish a new connection: [Errno -2] Name or service not known',))HTTPSConnectionPool(host='amlc-australiasoutheast-kv.vault.azure.net', port=443):

Do you see any additional information like above?

You can also check additional steps mentioned by Sina Salam to diagnose what is causing this. Thanks!!
Gupta, Shalu 30 Reputation points

2024-11-22T13:40:54.6266667+00:00
Hi @romungi-MSFT

Thank you for your suggestion and detailed explanation. I encountered the following error when checking the HTTPS connection:

File "/home/AzDevOps/.local/lib/python3.11/site-packages/azure/ai/ml/_artifacts/_blob_storage_helper.py", line 175, in check_blob_exists properties = blob_client.get_blob_properties() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ azure.core.exceptions.ServiceRequestError: <urllib3.connection.HTTPSConnection object at 0x7509b92b5dd0>: Failed to establish a new connection: [Errno -2] Name or service not known

This seems similar to the example you mentioned. Could you please confirm if this is related to a missing DNS configuration or network setup, or if there are any specific steps I should follow to resolve this?

Thank you again for your help!

1 answer

Sina Salam 12,976 Reputation points

2024-11-22T12:30:55.59+00:00
Hello Gupta, Shalu,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

I understand that you are having issue while deploying an Azure Machine Learning setup with private endpoints.

Most of all, the error you got, "Failed to establish a new connection: [Errno -2] Name or service not known," typically points to DNS resolution issues. However, the lists of a few steps below and links associated provide more steps and reading for better understand of related error on how you can resolve the issue.

Your DNS settings should be correctly configured, if you're using Azure DNS, verify that the private DNS zones for privatelink.blob.core.windows.net and privatelink.file.core.windows.net are correctly linked to your virtual network - and if you're using a custom DNS solution, make sure it can resolve the necessary FQDNs. You can test this by running nslookup <fqdn> from a VM within the same virtual network. - https://learn.microsoft.com/en-us/azure/machine-learning/how-to-troubleshoot-secure-connection-workspace?view=azureml-api-2

Double-check your private endpoints is associated with the appropriate resources and recreating the private endpoints can resolve configuration issues. - https://learn.microsoft.com/en-us/azure/machine-learning/how-to-configure-private-link?view=azureml-api-2

Verify that your NSGs allow traffic to and from the necessary services.- https://learn.microsoft.com/en-us/azure/machine-learning/how-to-configure-private-link?view=azureml-api-2

Should there be any proxy, it might be interfering with the connection. Try disabling the proxy temporarily to see if it resolves the issue. - https://learn.microsoft.com/en-us/azure/machine-learning/how-to-troubleshoot-secure-connection-workspace?view=azureml-api-2

Use the Azure Machine Learning workspace diagnostic API to identify any configuration problems with your workspace. - https://learn.microsoft.com/en-us/azure/machine-learning/how-to-troubleshoot-secure-connection-workspace?view=azureml-api-2

You've mentioned that the roles are configured, but it's worth double-checking that there are no missing permissions. - https://learn.microsoft.com/en-us/azure/machine-learning/how-to-configure-private-link?view=azureml-api-2.

The above are areas the which the error might be troubleshoot. Except there is general (temporary) downtime and/or IP Address issue.

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.
Please sign in to rate this answer.
Gupta, Shalu 30 Reputation points

2024-11-22T13:40:47.3433333+00:00

Hi @Sina Salem,

Thank you for your troubleshooting guide. I reviewed everything again and found one issue while running the workspace diagnostics:

ImageBuildComputeNotValid: "If the Container Registry is behind the virtual network, Container Registry cannot build your image. Set the imageBuildCompute property to build your image."

I am unsure why this error occurs because the Container Registry associated with the Azure ML workspace is configured with a private endpoint and resides in the same virtual network as the workspace.

Could you please help me understand how to fix this issue? All the other steps you mentioned seem to be working correctly.

Thank you again for your help!

Sina Salam 12,976 Reputation points

2024-11-22T14:12:57.7033333+00:00

Hello

Thank you for your feedback.

Well done for taking time to review the answer.

Regarding your new error, the error ImageBuildComputeNotValid typically occurs because the Azure Container Registry (ACR) cannot build the image when it's behind a virtual network is a common issue.

To fix this:

All that I have mentioned above still related to this issue and they are best way to troubleshoot it. The major addition will the followings:

You need to specify a compute target for building your images by using Azure CLI to run the following command and set the imageBuildCompute property:
az ml workspace update --name <your-workspace-name> --resource-group <your-resource-group> --image-build-compute <your-compute-cluster>
Replace the tags in the code <---- > with your actual workspace name, resource group, and compute cluster name. - https://techcommunity.microsoft.com/blog/fasttrackforazureblog/how-to-build-an-environment-when-your-azure-ml-workspace-is-behind-a-vnet/3657699

The ACR should be accessible from the compute target you specified, which implies that your virtual network and private endpoints are correctly configured. Try to use the az acr check-health command to diagnose network connectivity issues: az acr check-health --name <your-registry-name> --vnet

If you have a firewall or proxy in place, make sure that it is not blocking access to the ACR. You might need to configure firewall rules to allow traffic to the ACR's endpoints. - https://learn.microsoft.com/en-us/azure/container-registry/container-registry-troubleshoot-access

Every other thing will be in the range of ACR, NSG rules and DNS resolution. You check all the links provided above for more details.

Gupta, Shalu 30 Reputation points

2024-11-25T05:41:51.9533333+00:00

Hi @Sina Salem,

Thank you so much for your guidance and detailed troubleshooting steps. While reviewing and applying the suggestions you provided, I realized that my actual issue is slightly different and relates to the creation of a pipeline endpoint in Azure ML Studio.

Here’s a brief summary of the challenges I’m facing:

Pipeline Steps Issue:

When I attempt to create a pipeline endpoint, it fails during the steps for registering test data and training, specifically due to storage-related issues.

The system throws errors indicating it cannot create the required container or fetch data from it.

Dataset Creation Error:

While trying to create a dataset using the following function:
create_ml_client.data.create_or_update(aml_dataset)
It throws the following error: Error in creating data asset with Exception: <urllib3.connection.HTTPSConnection object at 0x7fcb17495fd0>: Failed to establish a new connection: [Errno -2] Name or service not known 3.Current Setup: The Container Registry is already configured with a private endpoint and resides in the same virtual network as the workspace, so I am unsure why this issue persists. It appears to be related to DNS or network configuration preventing proper access. Could you please provide further insights or guidance on how to address this issue? Your initial suggestions have been very helpful for narrowing down potential causes, but this specific issue remains unresolved. Thank you again for your help and support!

Sina Salam 12,976 Reputation points

2024-11-25T09:43:54.0466667+00:00

Hello Gupta, Shalu,

Thank you for more clarity and feedback.

The problem is not just inside the pipeline as it looks, if you can follow the below steps and the links provided you should be able identify the issue and troubleshoot it.

Storage-Related Issues. Main root cause to the pipeline fails due to DNS or networking misconfiguration preventing the AML workspace from accessing the storage account or creating necessary containers. So, make sure private DNS zones (privatelink.blob.core.windows.net) are correctly linked to the workspace's virtual network. Test DNS resolution for storage endpoints using bash command: nslookup <storage-account-name>.blob.core.windows.net Also, confirm that NSGs allow traffic between the workspace and the storage account and permanent fix to this is to use a diagnostic tool like Azure Network Watcher to ensure continuous connectivity between the workspace and storage endpoints.

Dataset Creation Error. The main root cause should the function create_ml_client.data.create_or_update() fails because the workspace cannot resolve or access the storage account. So, validate managed identity permissions:

Confirm the AML workspace's managed identity has at least Storage Blob Data Contributor role on the storage account.

If using custom DNS, configure conditional forwarding to Azure DNS for resolving Azure service FQDNs.

Use the Azure CLI to test storage account access using bash command:
`az storage blob list --account-name <storage-account-name> --container-name <container-name>`
Best permanent fix is to automate periodic DNS resolution checks for the workspace's storage and ACR endpoints to ensure no configuration drift.

Container Registry and Compute Target. The main root cause is the Container Registry (ACR) cannot be accessed or the workspace lacks a valid compute target for image building.

Run ACR health checks using bash: az acr check-health --name <acr-name> --vnet

Confirm the workspace's compute target has network access to ACR and the storage account. Update the workspace to use a valid compute target using bash: az ml workspace update --name <workspace-name> --resource-group <resource-group> --image-build-compute <compute-cluster-name> The permanent fix is to regularly validate ACR connectivity and workspace configurations using Azure Monitor or custom health checks.

The Pipeline Endpoint Creation. The root cause should be the network restrictions or improper configuration prevent pipeline steps from accessing necessary resources. What you need to do is to use diagnostic APIs to verify the pipeline environment using bash: az ml workspace diagnostics show --name <workspace-name> --resource-group <resource-group>

Enable diagnostic logs for AML and storage accounts to identify specific causes of pipeline failures.

Check whether required compute targets are attached and accessible. The permanent fix is to automate periodic validation of pipeline dependencies using Azure CLI or SDK scripts

Troubleshooting Links:

https://learn.microsoft.com/en-us/azure/machine-learning/how-to-configure-private-link?view=azureml-api-2

https://learn.microsoft.com/en-us/azure/machine-learning/how-to-troubleshoot-datasets

https://learn.microsoft.com/en-us/azure/machine-learning/how-to-troubleshoot-secure-connection-workspace?view=azureml-api-2

Gupta, Shalu 30 Reputation points

2024-11-25T13:01:50.8533333+00:00

Hi @Sina Salam
Thank you for providing detailed feedback and resources. I really appreciate the effort and the time you’ve taken to share these troubleshooting steps.

I’ve gone through the steps you mentioned, including DNS validation, managed identity permissions, ACR connectivity checks, and the pipeline diagnostic commands. Unfortunately, these didn’t resolve the issue in my scenario. The pipeline still fails at the endpoint creation step, and the _SYS private endpoints remain unclear in terms of how they are provisioned or configured via Bicep.

I’ll continue exploring the problem further and may raise it on Microsoft Q&A or with Azure support if needed. Thanks again for your support and insights!

romungi-MSFT 47,206 Reputation points Microsoft Employee

2024-11-25T14:09:27.0433333+00:00

Gupta, Shalu Thanks for checking the details. It seems like the DNS configuration you mentioned is also in place based on your above comment. Since this is a particular setup issue, I would encourage to raise a support case to have a support engineer debug this in your subscription. Thanks!!

Sina Salam Thanks for your troubleshooting steps.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Azure Machine Learning Endpoint Deployment

1 answer

Your answer