My Azure Container Instance is stuck "waiting" state during Rebuild. Can you tell us why?

Saša 1

Hello,

we try to rebuild containers for our app, but the containers are stuck in “waiting” without useful log messages. Is this a network issue. The build works fine locally.

The restart count seems to increase. Any idea how to debug or fix this?

All the best,

Saša

1 answer

deherman-MSFT 37,416 Reputation points Microsoft Employee

2024-08-23T21:34:15.4066667+00:00
@Saša

There a few things you can check to try and diagnose and resolve this.

Check Container Logs:

Retrieve the logs for the container instance to identify any errors or issues. You can use the Azure CLI command:

az container logs --resource-group <resource-group-name> --name <container-name>

Restart the Container: Sometimes, simply stopping and starting the container instance can resolve the issue. You can do this via the Azure Portal or using the Azure CLI:

az container stop --resource-group <resource-group-name> --name <container-name> az container start --resource-group <resource-group-name> --name <container-name>

Network Configuration: When container groups are deployed in virtual networks, port 19390 is required to connect to Container Instances from the Azure portal.

If you are still unable to resolve the issue, please open a support request so our engineers can troubleshoot this issue with you directly.

If you still have questions, please let us know in the "comments" and we would be happy to help you. Comment is the fastest way of notifying the experts.

If the answer has been helpful, we appreciate hearing from you and would love to help others who may have the same question. Accepting answers helps increase visibility of this question for other members of the Microsoft Q&A community.

Thank you for helping to improve Microsoft Q&A!
Please sign in to rate this answer.
Saša 1 Reputation point

2024-08-26T15:57:23.1133333+00:00

@deherman-MSFT

I restarted the containers without luck. I also deleted the repositories in my ACR and all containers.

Before I show you the logs - here is some context.

I am building 2 containers. One deploys a Weaviate DB and the other a vectorization service that connects to a Blob Storage and downloads a zipped file and data. The data is then used by the unzipped model.

The code works fine if I run it locally without docker and if I build containers. It fails on Azure.

I am using a Service Principal to connect to my data. The SP is a Contributer in my Subscription.

Here are my logs. These lines of code repeat over and over. I guess this is when the container restarts.

INFO:azure.core.pipeline.policies.http_logging_policy:Request URL: 'https://xxxx.blob.core.windows.net/data/models/clip-ViT-B-32.zip' Request method: 'GET' Request headers: 'x-ms-range': 'REDACTED' 'If-Match': '"0x8DCC383518A556C"' 'x-ms-version': 'REDACTED' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.11.7 (Linux-6.1.91.1-microsoft-standard-x86_64-with-glibc2.36)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': '20421ebe-63a8-11ef-9e1e-00155d16f463' 'Authorization': 'REDACTED' No body was attached to the request INFO:azure.core.pipeline.policies.http_logging_policy:Response status: 206 Response headers: 'Content-Length': '4194304' 'Content-Type': 'application/zip' 'Content-Range': 'REDACTED' 'Last-Modified': 'Fri, 23 Aug 2024 14:53:15 GMT' 'Accept-Ranges': 'REDACTED' 'ETag': '"0x8DCC383518A556C"' 'Server': 'Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0' 'x-ms-request-id': '2119e45e-801e-0038-23b4-f70064000000' 'x-ms-client-request-id': '20421ebe-63a8-11ef-9e1e-xxxx ' 'x-ms-version': 'REDACTED' 'x-ms-creation-time': 'REDACTED' 'x-ms-lease-status': 'REDACTED' 'x-ms-lease-state': 'REDACTED' 'x-ms-blob-type': 'REDACTED' 'x-ms-server-encrypted': 'REDACTED' 'Date': 'Mon, 26 Aug 2024 12:38:40 GMT' INFO:azure.core.pipeline.policies.http_logging_policy:Request URL: 'https:// xxxx .blob.core.windows.net/data/models/clip-ViT-B-32.zip' Request method: 'GET' Request headers: 'x-ms-range': 'REDACTED' 'If-Match': '"0x8DCC383518A556C"' 'x-ms-version': 'REDACTED' 'Accept': 'application/xml' 'User-Agent': 'azsdk-python-storage-blob/12.19.0 Python/3.11.7 (Linux-6.1.91.1-microsoft-standard-x86_64-with-glibc2.36)' 'x-ms-date': 'REDACTED' 'x-ms-client-request-id': '204d93a2-63a8-11ef-9e1e-xxxx ' 'Authorization': 'REDACTED' No body was attached to the request

I also get this line when running my code locally and it works fine.

No body was attached to the request

Whe you say "open a support request" are you talking about the Azure support?

deherman-MSFT 37,416 Reputation points Microsoft Employee

2024-08-26T16:52:46.4933333+00:00

You say you have Contributor role, but that does not grant access to the data. Can you double check that you have permissions to download the file?
https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles

Yes, I recommend opening an Azure Support Request so they can troubleshoot the issue directly.

Saša 1 Reputation point

2024-08-26T19:44:06.28+00:00

I do have the permissons. As I said the code works locally and in a docker container but no on Azure. I will write the Azure Support
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

My Azure Container Instance is stuck "waiting" state during Rebuild. Can you tell us why?

1 answer

Your answer