Create an alert when pod get stuck in ContainerCreating status for long time

Question

Is there a way to set up alert when pods are stuck in the "ContainerCreating" state for an abnormal amount of time? Currently my AKS logs go to Log Analytics Workspace. Currently, logs are enabled for Kubernetes Controller Manager, API and Kubernetes Cluster Autoscaler, also all metrics (see screenshot). I need help find out the proper way or best approach for this. Thanks.

Answer 1

Hi Scion,

Welcome to the Microsoft Q&A Platform! Thank you for asking your question here.

Yes, there is a way to set up alert when pods are stuck in the 'ContainerCreating' state for an amount of time. You can set up custom alerts rule in the Azure monitor.

Please try below steps:Go to Azure portal, then open Monitor from search, click on Alerts. Click on create Alert Rule then select Resource and resource type as Kubernetes services, then select AKS cluster and click on done. In the next tab, configure Alert rule conditions, choose custom log search and use a Kusto query to get alerts when pods are stuck. Next configure each tab. For more detailed information, please check Configure the scope of the alert rule, Configure actions

You can try editing this query or try custom query Container logs for any failed pod in a specific namespace

Please check these below documents for detailed information and steps to configure: Create log search alerts from Container insights

To know more about Azure Monitor Alerts, please visit What are Azure Monitor alerts?

If you have any further queries, please do let us know. If the answer is helpful, please click "Accept Answer" and "Upvote it."