Azure Container App Down

Question

We had two app containers configured on the same environment go down suddenly. They both pull from the same Azure Container Registry.

These apps have been deployed many times and run without any configuration changes for months prior to going down. Initially, logs indicated they could not find resources on the network (ACR and databases).

Container Environment is configured as Consumption-based.

Initial Issue:

Both containers and databases are on on the same virtual network. The containers share a subnet and databases share a subnet. There have been no network configuration changes. The databases are up. The ACR is up and accessible from other resources. The problem persists.

1. They are configured using service connectors for the databases.
2. The only oddity I can see is that they experienced high network traffic starting on 5 Feb. My only thought is perhaps hitting a network limit/quota which caused the containers to be blocked on that network? I don't know that to be true (or even the case), but I have tried just everything short of creating a new environment on a new network.

Attempts to Fix/Trace:

When we relax network configurations just to get them running before applying a fix, the containers find the resources (despite originally not having an issue) but hang on activation and never start. They eventually timeout. Both containers experience this timeout behavior.

1. How do we get through stuck containers just to get them running? They are in a perpetual "Activating" state.
2. Is there any way to determine what caused the issues considering the configurations were unchanged prior to the breakage for months? Both containers went down simultaneously and remain down.

We've sifted through countless logs, events, and alerts from all angles and cannot find neither a cause nor fix.

Answer 1

Hello Greg L,

Thank you for posting your question in the Microsoft Q&A forum.

As I am understanding sudden failure of both app containers on your side, combined with the symptoms you’ve described (network resource issues, activation timeouts, and high network traffic), suggests a complex issue that could stem from a combination of network, resource, or platform-level problems.

By systematically addressing the areas below, you should be able to identify the root cause and resolve the issue so verify and confirm below configurations:

1. Investigate network quotas and limits (e.g., Network In and Network Out etc.) – Useful link to review - https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/azure-subscription-service-limits
2. Check container activation logs and resource allocation. (stdout/stderr logs, CPU and memory allocation etc.) Useful link to review - https://learn.microsoft.com/en-us/azure/container-instances/container-instances-troubleshooting
3. Validate service connectors and dependencies. (database connectivity and ACR access) Useful link to review - https://learn.microsoft.com/en-us/azure/service-connector/overview
4. Review platform-level issues and Azure status. Useful link to review - https://learn.microsoft.com/en-us/azure/service-health/service-health-overview
5. Troubleshoot VNet configuration (NSG, DNS, private endpoints). Useful link to review - https://learn.microsoft.com/en-us/azure/private-link/private-endpoint-overview
6. Implement temporary workarounds (new environment, reduce traffic). Useful link to review - https://learn.microsoft.com/en-us/azure/app-service/environment/creation
7. Engage Azure Support if the issue persists. Useful link to review - https://learn.microsoft.com/en-us/azure/azure-portal/supportability/how-to-create-azure-support-request
8. How do we get through stuck containers just to get them running? They are in a perpetual "Activating" state. - It typically indicates an issue with initialization, resource allocation, or dependencies.
9. Is there any way to determine what caused the issues considering the configurations were unchanged prior to the breakage for months? Both containers went down simultaneously and remain down. - Determining the root cause of an issue where both containers went down simultaneously and remain down despite no configuration changes for months requires a systematic investigation.

If the above answer helped, please do not forget to "Accept Answer" as this may help other community members to refer the info if facing a similar issue.