How will Azure App Service behave if a running App Service cannot reach a private registry?

Question

Hello,

We had several customers experience an outage when a private registry they were using was down. The log entry in AppService was:

2024-09-04T12:12:25.232Z INFO - Pulling image: registry.XXX.com/XXX/release:3.2.34494

2024-09-04T12:12:25.491Z ERROR - DockerApiException: Docker API responded with status code=InternalServerError, response={"message":"Get "https://registry.XXX.com/v2/": dial tcp 3.83.225.2:443: connect: connection refused"}

2024-09-04T12:12:25.493Z ERROR - Pulling docker image registry.XXX.com/XXX/release:3.2.34494 failed:

2024-09-04T12:12:26.301Z ERROR - DockerApiException: Docker API responded with status code=InternalServerError, response={"message":"Get "https://registry.XXX.com/v2/": dial tcp 3.83.225.2:443: connect: connection refused"}

2024-09-04T12:12:26.302Z WARN - Image pull failed. Defaulting to local copy if present.

2024-09-04T12:12:26.305Z ERROR - Image pull failed: Verify docker image configuration and credentials (if using private repository) 2024-09-04T12:12:26.308Z INFO - Stopping site XXX-prod-001 because it failed during startup.

Questions:

Q1. How often does a running App Service re-pull an image from registry?

Q2. "Defaulting to a local copy if present" - I'm curious why this didn't happen, and if we are missing a setting somewhere to keep a local copy.

Q3. Is the behavior of the above two questions documented somewhere?

Thanks!

Chris Justus

Answer 1

Hello @Chris Justus , we're are sorry to hear your customers experienced this issue. Below are answers to your questions:

Q1. How often does a running App Service re-pull an image from the registry?

By default, a running App Service does not automatically re-pull an image from the registry. It only pulls the image once during deployment. However, there are a few scenarios where a re-pull could happen:

• If you manually redeploy your app through the Azure portal, CLI, or other tools, it will trigger a new image pull.
• If your deployment configuration specifies a different image tag or registry, it might trigger a re-pull.
• If you've set up webhooks to automatically trigger deployments when there's a change in your image repository (e.g., GitHub push), each push could trigger a re-pull.
• If your App Service has a startup probe configured and it fails repeatedly, Azure might attempt to restart the container, which could involve re-pulling the image.

Q2. "Defaulting to a local copy if present" - I'm curious why this didn't happen, and if we are missing a setting somewhere to keep a local copy.

The log message "Defaulting to a local copy if present" indicates that App Service attempted to use a previously downloaded image if it was available. However, in your case, it seems no local copy was found. Here are some reasons why a local copy might not have been available:

• If this was the first deployment of the image for this specific App Service instance, there wouldn't be a local copy yet.
• Docker uses image layer caching to optimize image downloads. If the base layers of your image haven't changed, they might be pulled from the local Docker daemon cache instead of the registry. However, the specific image version (tag) still matters.
• If your App Service instance is low on disk space, it might not have enough space to store a local copy of the image.

There isn't a specific setting in App Service to force keeping a local copy of the image. However, Docker itself offers features like image caching that can help ensure some layers are readily available locally.

Q3. Is the behavior of the above two questions documented somewhere?

Yes, the behavior you described is documented in several resources:

• Azure App Service container images
• Troubleshooting Guide: Resolving Azure App Services Image Pull Issues from Azure Container Registry

Hope this helps. Let us know if you have further questions

Best,

Grace