Azure Container Registry
An Azure service that provides a registry of Docker and Open Container Initiative images.
465 questions
Container App Activation Failed without error, checked everything, spent 2 days by now, please help
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(3.2, 31%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EFT%3C/text%3E%3C/svg%3E)
Filipp Trigub
0
Reputation points
Hello,
I got 3 container apps running in a container app env with workload profiles Consumption and D4. The Consumption type apps startup normally. The D4 type app failsto start up, but only inconsistently. These 3 apps are run in 3 deployment envs (dev sandbox , prod).
The image certainly works.
Failure to deploy a new revision via CICD occurs consistently for the sandbox environment, but never for dev.
Monitoring shows not errors, no alerts, no warnings. CPU and memory are well within environment bounds, but usually maxed out for the app itself (4 Ghz, 16 GB, max at D4).
Manual redeployment works...sometimes...
I genuinly have no idea what the issue is.
LOGS
No Console Logs are generate in case of a fail. Here the system logs:
11/18/2024, 8:43:03.790 PM
Stopping container file-reader-app-sandbox
11/18/2024, 8:43:03.739 PM
Deactivated apps/v1.Deployment k8se-apps/file-reader-app-sandbox--176otp8 from 1 to 0
11/18/2024, 8:42:49.647 PM
Started container file-reader-app-sandbox
11/18/2024, 8:42:49.571 PM
Created container file-reader-app-sandbox
11/18/2024, 8:42:49.560 PM
Successfully pulled image "snaacr.azurecr.io/file-reader-app:2024-11-18-18-30-46-2292e694" in 2m20.958s (2m20.958s including waiting)
11/18/2024, 8:40:28.602 PM
Pulling image "snaacr.azurecr.io/file-reader-app:2024-11-18-18-30-46-2292e694"
11/18/2024, 8:40:18.099 PM
Replica 'file-reader-app-sandbox--176otp8-58485b9fc7-wqxtk' has been scheduled to run on a node.
11/18/2024, 8:38:20.407 PM
Scaling up infrastructure to accommodate replica 'file-reader-app-sandbox--176otp8-58485b9fc7-wqxtk'
11/18/2024, 8:38:03.671 PM
KEDA is starting a watch for revision 'file-reader-app-sandbox--176otp8' to monitor scale operations for this revision
11/18/2024, 8:38:03.647 PM
Scaler cpu is built.
11/18/2024, 8:38:03.642 PM
Scaler external-push is built.
11/18/2024, 8:38:03.635 PM
Scaler external-push is built.
11/18/2024, 8:38:03.553 PM
Successfully provisioned revision 'file-reader-app-sandbox--176otp8'
11/18/2024, 8:37:44.300 PM
ScaledObject was removed from KEDA watch and would not be auto-scaled. Please check https://learn.microsoft.com/en-us/azure/container-apps/dapr-overview
Here the system logs of a healthy revision (exactly the same specs)
11/18/2024, 8:43:13.510 PM
Readiness probe failed: Get "http://100.100.0.248:9000/readiness": dial tcp 100.100.0.248:9000: connect: connection refused
11/18/2024, 8:43:03.343 PM
Stopping container file-reader-app-sandbox
11/18/2024, 8:43:03.285 PM
Deactivated apps/v1.Deployment k8se-apps/file-reader-app-sandbox--h3ekw97 from 1 to 0
11/18/2024, 8:38:09.475 PM
Started container file-reader-app-sandbox
11/18/2024, 8:38:09.405 PM
Created container file-reader-app-sandbox
11/18/2024, 8:38:09.399 PM
Successfully pulled image "snaacr.azurecr.io/file-reader-app:2024-11-18-17-20-58-91551927" in 580ms (580ms including waiting)
11/18/2024, 8:38:08.817 PM
Pulling image "snaacr.azurecr.io/file-reader-app:2024-11-18-17-20-58-91551927"
11/18/2024, 8:38:03.209 PM
KEDA is starting a watch for revision 'file-reader-app-sandbox--h3ekw97' to monitor scale operations for this revision
11/18/2024, 8:38:03.204 PM
Replica 'file-reader-app-sandbox--h3ekw97-6b87844db7-8tx58' has been scheduled to run on a node.
11/18/2024, 8:38:03.188 PM
Scaler external-push is built.
11/18/2024, 8:38:03.177 PM
Scaler cpu is built.
11/18/2024, 8:38:03.171 PM
Scaler external-push is built.
11/18/2024, 8:37:44.306 PM
ScaledObject was removed from KEDA watch and would not be auto-scaled. Please check https://learn.microsoft.com/en-us/azure/container-apps/dapr-overview
11/18/2024, 8:37:44.277 PM
KEDA is stopping the watch for revision 'file-reader-app-sandbox--h3ekw97' to monitor scale operations for this revision
{count} votes
-
Ryan Hill 28,726 Reputation points Microsoft Employee2024-11-18T22:42:55.8533333+00:00 @Filipp Trigub can you elaborate more on the following?
but usually maxed out for the app itself (4 Ghz, 16 GB, max at D4)
Are you saying here that app maxes the D4 profile at startup? Are the three apps the same app being deployed to each individual environment? If yes, does the app fail to start across all three environments or just dev? If it's the same app between Consumption and D4, how does the resource allocation look? Are there any scaled events on the Consumption profile at startup?
-
Filipp Trigub 0 Reputation points
2024-11-19T08:53:55.9266667+00:00 Hi, sure, thanks for the quick reply!
Are you saying here that app maxes the D4 profile at startup?
The workload profile is created with max instances set to 10. There is only 1 app starting in this profile. It does not max out the total max cpu count or total max memory allocated to the entire profile. That said the container is set to run at max cpu / memory of D4 (4 cores, 16 GB RAM).Are the three apps the same app being deployed to each individual environment?
No, these are 3 different apps. The 2 other ones (frontend, backend) are far less resource-intensive and run on Consumption.If yes, does the app fail to start across all three environments or just dev?
Only file reader (the D4 app) fails and only on sandbox and prod.
My current guess is that it might have something to do with the auth. The resources have been created an modified with my user acc via terraform, but the deployment runs via an SP. Will keep you updated. Moreover I was able to consistently run a container that I deployed manually via the portal, but the auto-deployed one failed, even if I would restart it manually. Both containers had the exact same spects with only some env vars differing.
-
Filipp Trigub 0 Reputation points
2024-11-19T09:28:35.5166667+00:00 Sporadically I am getting this system log mssage:
0/4 nodes are available: 1 node(s) had untolerated taint {k8se.microsoft.com/SwiftSetupOnly: true}, 1 node(s) had untolerated taint {virtual-kubelet.io/provider: legion}, 2 node(s) didn't match Pod's node affinity/selector. preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling.
I was not able to confirm that it is always present in case of a failed Activation. Nevertheless, its cryptic enough to make me think.
This is something related to KEDA, but documentation is sparse, so I hae no idea, what it would mean.
-
Filipp Trigub 0 Reputation points
2024-11-19T09:51:22.62+00:00 Another observation:
Revision deployed via CICD fails.
Redploying it manually with changes to the health probes fails.
Redeploying it manually with 1 additional dummy env var, succeeds. -
Filipp Trigub 0 Reputation points
2024-11-19T11:53:07.4233333+00:00 Dev environment also failed with a similar message
0/5 nodes are available: 1 node(s) had untolerated taint {ToBeDeletedByClusterAutoscaler: 1732011474}, 1 node(s) had untolerated taint {virtual-kubelet.io/provider: legion}, 3 node(s) didn't match Pod's node affinity/selector. preemption: 0/5 nodes are available: 5 Preemption is not helpful for scheduling.
As these are simple container apps, the cluster is not listed. Is there a way to check the nodes?
-
Ryan Hill 28,726 Reputation points Microsoft Employee
2024-11-21T16:43:45.8633333+00:00 Thanks @Filipp Trigub for the additional insights. I'll need work more closely with you in order to determine the root cause here. Please respond to the private message so I can gather those details.
Sign in to comment
Sign in to answer