Cluster cant schedule new pods.

Question

Cluster cant schedule new pods.

Velizar Danev 0

Kubernetes Cluster cannot issue new pods. All currently issued ones are active, but as soon as you try to restart a pod or create a new one its stuck in pending mode.No Recent events events or conditions in the pod/deployment details.
All nodes appears as active with no taints and not resource locked.
Only error i find is in the events and it states |ContainerRuntimeProblem True 12 days ago crictl -t 60s pods --latest failed!|| | -------- | -------- |

Velizar Danev 0 Reputation points

2025-03-06T10:11:42.9566667+00:00

This is the only error i see in the events of the nodes
ContainerRuntimeIsDown] crictl -t 60s pods --latest failed!
component: container-runtime-custom-plugin-monitor

host: aks-nodepool1-89500097-0

type: Warning
Ketan Dani 0 Reputation points

2025-03-10T08:59:25.67+00:00

We have 2 clusters on AKS (v1.18.14) and we are facing the same issue since 5th March 2025. The scheduler seems to be in a unhealthy state due to which pods are not allocated to node. This can be verified by running below command -
kubectl get componentstatuses

As a temporary solution, we have specified 'nodeName' in our deployment yamls since this affects our production.

1 answer

Your answer

Velizar Danev 0 Reputation points

2025-03-06T10:11:42.9566667+00:00

This is the only error i see in the events of the nodes
ContainerRuntimeIsDown] crictl -t 60s pods --latest failed!
component: container-runtime-custom-plugin-monitor

host: aks-nodepool1-89500097-0

type: Warning
Ketan Dani 0 Reputation points

2025-03-10T08:59:25.67+00:00

We have 2 clusters on AKS (v1.18.14) and we are facing the same issue since 5th March 2025. The scheduler seems to be in a unhealthy state due to which pods are not allocated to node. This can be verified by running below command -
kubectl get componentstatuses

As a temporary solution, we have specified 'nodeName' in our deployment yamls since this affects our production.

Answer 1

LISBOA 80 Microsoft Employee

Hello Velizar Danev

Thank you for reach out to us!

Could you please check the events for the nodes, using kubectl describe nodes <node-name>

https://kubernetes.io/docs/reference/kubectl/generated/kubectl_describe/

Try to search for the Node where the pods should be scheduled.

For the pods in pending, please use kubectl describe pod <pod-name> and check tha status/events inside the pod.

If possible create a new nodepool, and move your workload for the new nodepool, observe also if you have a Vnet/Subnet with a size that accomodate the number of IP's needed.

https://learn.microsoft.com/en-us/azure/aks/concepts-network-ip-address-planning

Also share detaisl about your AKS cluster, such as version, size, and so on.

If the answer has been helpful, we appreciate hearing from you and would love to help others who may have the same question.

Accepting answers helps increase visibility of this question for other members of the Microsoft Q&A community.

Thank you for helping to improve Microsoft Q&A!

Nagarjuna Reddy Yanna 255 Reputation points Microsoft External Staff

2025-03-07T08:28:15.0633333+00:00

Hi Velizar Danev,

Thank you for reaching out to us on the Microsoft Q&A forum

Just checking in to see if you had a chance to review the answer posted by LISBOA on your question. Please let us know if it was helpful and feel free to reach out if you have any further queries.

If you found the information useful, please click "Upvote" on the post to let us know.

Thank You
Velizar Danev 0 Reputation points

2025-03-07T09:06:17.74+00:00

DurationInSeconds: -1. Virtual machine is going to be restarted as requested by authorized user.

Normal Starting 10m kubelet Starting kubelet.

Normal NodeHasSufficientMemory 10m (x2 over 10m) kubelet Node aks-nodepool1-89500097-4 status is now: NodeHasSufficientMemory

Normal NodeHasNoDiskPressure 10m (x2 over 10m) kubelet Node aks-nodepool1-89500097-4 status is now: NodeHasNoDiskPressure

Normal NodeHasSufficientPID 10m (x2 over 10m) kubelet Node aks-nodepool1-89500097-4 status is now: NodeHasSufficientPID

Warning Rebooted 10m kubelet Node aks-nodepool1-89500097-4 has been rebooted, boot id: fc141968-2f12-4d7c-9b42-c404b61bfc0d

Normal NodeAllocatableEnforced 10m kubelet Updated Node Allocatable limit across pods

Normal NodeReady 10m kubelet Node aks-nodepool1-89500097-4 status is now: NodeReady

Normal NoVMEventScheduled 10m custom-scheduledevents-consolidated-condition-plugin-monitor Node condition VMEventScheduled is now: False, reason: NoVMEventScheduled, message: "VM has no scheduled event"

Normal KubeletIsUp 10m kubelet-custom-plugin-monitor Node condition KubeletProblem is now: False, reason: KubeletIsUp, message: "kubelet service is up"

Warning ContainerRuntimeIsDown 67s (x20 over 10m) container-runtime-custom-plugin-monitor crictl -t 60s pods --latest failed!

those are the events.

I can't seem to create a new nodepool because the cluster is using availability sets.
I also tried increasing the current nodepool size but nothing happens.
The kubernetes version is 1.18.14.
I've already created a new cluster and started migrating the workloads.I only want to know why it happend, so i can prevent it on my other clusters.
Nagarjuna Reddy Yanna 255 Reputation points Microsoft External Staff

2025-03-07T18:21:13.63+00:00
Hi Velizar Danev,

We have noticed that answer provided by LISBOA was not helpful. We appreciate your feedback and are committed to improving your experience with the Q&A.

The information you have provided suggests that your AKS (Azure Kubernetes Service) node underwent a reboot, which may result from various factors. Below are some possible causes for the reboot, along with recommendations to mitigate similar occurrences in the future.

Check the Node's Container Runtime (e.g., containerd or Docker):

The container runtime issue is likely preventing new containers from starting or causing the existing ones to fail.

If you have direct access to the node (via SSH, for example), you can check the status of the container runtime:

# For containerd: systemctl status containerd # For Docker: systemctl status docker

Restart the container runtime if it's down:

sudo systemctl restart containerd # or docker

Check Node Resource Usage:

Even though the logs indicate no disk pressure or memory issues, it's still a good idea to monitor the system resources of the node to ensure that the container runtime has enough resources.

You can use top or htop to check CPU and memory usage on the node.

Verify kubelet and Docker/Containerd Logs:

Since kubelet is responsible for interacting with the container runtime, checking the logs for kubelet and the container runtime might provide additional information about what went wrong:

# For kubelet logs: journalctl -u kubelet # For containerd logs: journalctl -u containerd # or docker

Pod Scheduling and Container Restart:

If the container runtime comes back up, but you still see issues with containers not starting, manually restart the affected pods or nodes to trigger rescheduling and reinitialization:
kubectl get pods -o wide # Get a list of pods and nodes kubectl delete pod <pod_name> # Manually restart a pod

Check if the containers are scheduled and running on another node if available.

Verify AKS Node Health:

AKS handles much of the underlying infrastructure for you, so if this issue persists or happens frequently, it may indicate a larger issue with the AKS infrastructure or the node pool itself.

You can check the health of the node pool:
az aks nodepool show --resource-group <resource-group-name> --cluster-name <aks-cluster-name> --name <nodepool-name>

If you find that the node pool is unhealthy or experiencing recurring issues, consider scaling it up or out or recreating the node pool.

Check for Pending Kubernetes Events:

Use the kubectl describe node <node-name> command to check for any pending events that could provide additional insight into why the container runtime is down:
kubectl describe node aks-nodepool1-89500097-4

The main concern appears to be that the container runtime is not operational. Restarting the runtime or examining its configuration may rectify the situation. Should the issue continue, it is advisable to review the logs for additional insights, assess the status of the AKS node pool, and contemplate scaling or recreating the node pool if required.

Please find the below documentation for your reference.

https://learn.microsoft.com/en-us/azure/aks/create-node-pools

https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/windows/unexpected-vm-reboot-root-cause-analysis

https://kubernetes.io/releases/version-skew-policy/

If the information is helpful, please consider by clicking the "Upvote" on the post.

If you have any further queries, please let us know in the comment.

Thank you.
Velizar Danev 0 Reputation points

2025-03-07T19:43:57.9033333+00:00

Both the docker and containerd appears active and withot any errors there are no logs in kubelet journalctl
as far as the nodepool it now appears as failed
Provisioning state__

:

Failed



i tried to scale it from 5 to 10 but it cant issue new nodes i would suggest since its very old kubernetes version.
as far as replacing the nodepool this is the message i get
This cluster is using availability sets instead of scale sets for node pools. This limits the functionality available for managing node pools, such as upgrading Kubernetes versions for individual node pools and adding or removing node pools. To enable these features, create a new cluster with virtual machine scale sets enabled.
Nagarjuna Reddy Yanna 255 Reputation points Microsoft External Staff

2025-03-10T14:18:24.6933333+00:00
Hi Velizar Danev,

1.Nodepool Scaling Issue:

You attempted to increase the node pool from 5 to 10; however, the system was unable to provision new nodes. This issue may arise for several reasons, even when Docker and containerd are operational. It could be related to the configuration of the cluster or restrictions set by the existing setup.

2.Cluster Configuration - Availability Sets vs. Scale Sets:

Availability Sets: Availability Sets represent a traditional approach to structuring virtual machines (VMs) within Azure, designed to enhance high availability by distributing VMs across various fault domains.

Virtual Machine Scale Sets (VMSS):This represents a more recent and adaptable solution that facilitates automatic scaling and simplifies the management of node pools.

3.Provisioning State Failure:

The "Provisioning state: Failed" may result from an outdated configuration that does not accommodate recent scaling operations or upgrades to Kubernetes.

Upgrade the Cluster Setup to Use Scale Sets:

It is advisable to establish a new Kubernetes cluster that incorporates Virtual Machine Scale Sets (VMSS). This approach will provide the necessary flexibility to scale, upgrade, and manage node pools independently. After the new cluster with VMSS is set up, you can proceed to transfer workloads from the existing cluster to the new one.

Regarding the Kubernetes Version:

Given that the current Kubernetes version is outdated, it is prudent to upgrade the cluster to a version that is still supported. If the cluster is operating on a version prior to 1.18, an upgrade could address compatibility challenges and enhance scaling capabilities. However, since the existing cluster utilizes Availability Sets, upgrading individual node pools may not be practical unless a transition to VMSS is made.

6.Establish a New Cluster with VMSS:

In the Azure environment, you can initiate the creation of a new Azure Kubernetes Service (AKS) cluster with VMSS enabled. The steps involved include:

Setting up a new AKS cluster with VMSS functionality.

Configuring the node pools to utilize scale sets.

Transferring workloads, such as pods and deployments, from the old cluster to the new one using tools like kubectl, Helm, or through CI/CD pipelines.

Removing the old cluster after successful migration.

Investigate Azure Resource Constraints:

If scaling issues persist despite the appropriate configuration, it is essential to examine any Azure-related resource limitations, such as quotas or the availability of VM types in your region. Resource allocation restrictions within your Azure subscription may be hindering the scaling of nodes.

If the information is helpful, please consider by clicking the "Upvote" on the post.

If you have any further queries, please let us know in the comment.

Thank you.
Nagarjuna Reddy Yanna 255 Reputation points Microsoft External Staff

2025-03-12T03:11:08.6466667+00:00

Hi Velizar Danev,

Just checking in to see if you had a chance to review my comment on your question. Please let us know if it was helpful and feel free to reach out if you have any further queries.

If you found the information useful, please click "Upvote" on the post to let us know.

Thank You.
Andrew Harper 0 Reputation points

2025-03-12T23:06:26.7333333+00:00

@Ketan Dani Were you able to find a more permanent solution? Your answer was helpful!

I have a similar version to you and editing the nodeName does work as a temporary fix. it seems the fault lies on the Scheduler side in the Control Plane which is managed by AKS, but I am not sure if it is something we can fix from our side or if Azure is willing too look into such an old system.

Share via

Cluster cant schedule new pods.

1 answer

Your answer