Azure CI/CD Pipeline With ACR Private EndPoint

Question

Hi Everyone,

We have created an Azure DevOps pipeline and releases for CI/CD and we're using Azure container registry(ACR) for pulling/pushing the docker images, these pipelines/releases working and are able to pull/push the images without an issue as the Azure container registry(ACR) is public. But now our security team has asked us to make the ACR registry private, so we made ACR registry private using the private endpoint concept but now our pipelines/releases are failing as we made ARC private.

Answer 1

@Lokesh Anji

• Once the public network access is disabled, Instances of certain Azure services including Azure DevOps Services are currently unable to access the container registry.
• Private endpoints are not currently supported with Azure DevOps managed agents. You will need to use a self-hosted agent with network line of sight to the private endpoint.
• Access to Azure Container Registry (ACR) can be restricted to the virtual network using Private Endpoints. This will limit ACR exposure to public internet. Since private ACR is available only within the vnet, self-hosted devops agents comes to the rescue.

Configure the agent to access the private ACR: If you're using a self-hosted agent, ensure that the agent has access to the private ACR. You can do this by adding the agent's IP address to the allowed list in the ACR's firewall settings or by configuring the agent to use a private endpoint.

Reference documentation:

• https://techcommunity.microsoft.com/t5/fasttrack-for-azure/using-azure-devops-to-deploy-an-application-on-aks-private/ba-p/2029630
• https://learn.microsoft.com/en-us/azure/container-registry/container-registry-private-link

Answer 2

There are a few solutions to this issue:

1. allow all ip addresses for DevOps Agents in your ACR firewall rule. This implies that your ACR is still accessible via the internet. This can be done via a service tag: https://devblogs.microsoft.com/devops/azure-devops-service-tag-released/
2. allow the IP address for the currently used agent instance temporarily to the firewall of your ACR. This still allows public access to your ACR, but only for that one agent, and only during deployment. See https://build5nines.com/what-is-ip-address-of-azure-devops-build-agent-and-set-firewall-to-allow-it/
3. use a self-hosted agent (https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=azure-devops&tabs=browser)

The first solution is the easiest. However, ususally considered too insecure. The third option is the most secure. However, this option requires you to set-up a self-hosted agent and maintain it. Not maintaining it might also be a security risk. Best of both worlds would be to use the second approach, if allowed by your security team.

Answer 3

Has anything changed regarding this? We currently use the ACR firewall as a workaround, but it is extremely fragile. Our builds fail 30% of the time because the firewall changes often take too long to take effect or get confused when more than one build is running at the same time, and they conflict with each other. It's a terrible solution. Self-hosted agents are a non-starter for us. There has to be a way to allow cloud agents to work over a private link. What are the technical challenges to getting this to work?