This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(284.8, 39%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ED%3C/text%3E%3C/svg%3E)
I have a cluster which is made up of 3 node pools:
system - System Node Pool
project 1 - User Node Pool
project 2 - User Node Pool
On the System node pool, I have the taint set CriticalAddonsOnly:true:NoSchedule.
My issue is that for some reason, system pods, for example CoreDNS, Metric Server are all being schedule on project 1 & project 2 Nodes?
I have not used my capacity on the System Nodes either, there is still loads of space for more Pods.
Why is this happening and how can I stop this? I want system pods on the system node pool only.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(54.400000000000006, 73%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Hi devopsfj,
Welcome to the Microsoft Q&A Platform! Thank you for asking your question here.
We understand from your query that you are experiencing an issue where System pods are being scheduling into User Node Pools. I have shared troubleshooting steps that I felt will help resolve the issue you reported.
If the System node pool becomes unhealthy or unavailable, Kubernetes may schedule system pods on available user node pools. For system pods managed by DaemonSets, Kubernetes will schedule one pod per node across all nodes, including user pools.
You can Label System Node Pool Nodes with a unique label and Add Node Affinity to System Pods like CoreDNS and Metric Server. This ensures the system pods are preferred to be scheduled on the System node pool as long as there is capacity.
For example: label: node-role.kubernetes.io/system=true
kubectl label nodes <system-node-name> node-role.kubernetes.io/system=true
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: "node-role.kubernetes.io/system"
operator: In
values:
- "true"
See that User node pools should not tolerate the CriticalAddonsOnly=true:NoSchedule taint to prevent system pods from scheduling there. Review the node tolerations on user node pools to ensure that there are no tolerations for the CriticalAddonsOnly=true:NoSchedule taint that would allow system pods to bypass this restriction.
After checking with the above configurations, redeploy system pods to let Kubernetes reschedule them according to the updated rules.
If you still find any difficulties, please let me know I would like to work closer on this issue.
Thank you.
Hi @anashetty
Currently only the system node pool has the taint CriticalAddonsOnly=true:NoSchedule.
My user nodes do not that have this taint, yet system pods are being scheduled on user nodes?
I understand that some components like Kube Proxy, CSI Provider etc should have one pod on each node, however, core pods like Core DNS which only have two pods are not being scheduled on System Nodes... this doesn't seem correct?
How do you recommend adding these node labels to the Pods, will these not be removed when there is a patch to AKS?
Is there no way out the box without adding custom labels to schedule these system pods on the system node?
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(265.6, 22%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKJ%3C/text%3E%3C/svg%3E)
I have the same concern.
Seems like some pods that are considered to be system pods do not have affinity to the system nodes. Sometimes all replicas are deployed in system nodes but other times are being scheduled in user nodes as well. We have also noticed that in some cases (e.g., gatekeeper) the MS preconfigured replicas (2) are fully scheduled in user nodes.
In our case we have seen this behaviour with Gatekeeper, ama-logs, istio asm, metrics server and others. Worth mentioning that some of these are AKS add-ons that are just enabled (sometime from the portal) and MS provides the deployment configuration.
I'm interested to understand why MS do not consider affinity for these system pods.
Hi devopsfj,
Thanks for getting back with the detailed explanation. I have provided more troubleshooting steps that might help you.
Node labels added to Pods in Azure Kubernetes Service (AKS) may not persist through patches or upgrades. Using AKS-managed labels like kubernetes.azure.com/mode: system in conjunction with nodeSelector is a best practice for ensuring that system pods are scheduled on the appropriate nodes. This approach helps maintain the stability and performance of your AKS cluster, especially during upgrades and patches.
Try to enforce scheduling system-critical pods on your System node pool using AKS-provided node labels:
List the nodes in your cluster and check their labels: kubectl get nodes --show-labels The nodes in the System node pool should have the label: kubernetes.azure.com/mode=system
Modify the deployment specification of system-critical pods to include a nodeSelector that targets the System node pool. Locate the deployment of a system pod: kubectl get deployments -n kube-system Edit the deployment: kubectl edit deployment coredns -n kube-system This will open the deployment specification in your default editor. Add the nodeSelector: In the spec.template.spec section, add the nodeSelector targeting the kubernetes.azure.com/mode=system label:
spec:
template:
spec:
nodeSelector:
kubernetes.azure.com/mode: system
After this, save and exit the editor. This will update the deployment.
After updating the deployment, Kubernetes will apply the new nodeSelector, but existing pods may not immediately move to the System node pool. You can force them to reschedule: Delete existing pods for the deployment: kubectl delete pod -n kube-system -l k8s-app=kube-dns Kubernetes will automatically recreate the pods based on the updated deployment configuration. Now verify the pods are running on system node pool: kubectl get pods -n kube-system -o wide
Now Repeat this for Other System Pods to Modify the deployment specification of system-critical pods.
You can try using affinity rules. This method allows you to enforce scheduling preferences while providing fallback options if System nodes become unavailable: Define a preferredDuringSchedulingIgnoredDuringExecution rule: Update the system pod deployments to include a nodeAffinity specification.
spec:
template:
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: "kubernetes.azure.com/mode"
operator: "In"
values:
- system
Please let us know if you have any further queries. I’m happy to assist you further.
Thank you.
Hi Koury, Juan,
Welcome to the Microsoft Q&A Platform! Thank you for asking your question here.
Microsoft's design choice to not enforce pod affinity or anti-affinity for AKS system pods is driven by the need to prioritize flexibility and reliability in system workloads across node pools. By deploying system pods with tolerations for taints applied to system node pools, they can operate seamlessly without being impacted by specific scheduling constraints, allowing Kubernetes to balance workloads effectively during scaling, upgrades, or node failures. While Microsoft provides guidelines for configuring taints and tolerations to isolate workloads, there is no default enforcement for affinity on system pods.
Please let us know if you have any further queries. I’m happy to assist you further.
Hello @anashetty @Koury, Juan
I believe I may have found the issue to what was happening...
I believe my System Nodes did not have the accepted amount of CPU/Memory left to facilitate all system pods with the CriticalAddonsOnly:true:NoSchedule taint.
Once I increased my system nodes to have the recommended specs as stated here on MS Docs..
All of the system pods were schedules on the system nodes (apart from DaemonSets of course).I will monitor the pods over the next few days to ensure this is correct..
Hey devopsfj,
I have reposted your solution as an answer, Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", if you'd like to "Accept " the answer it will be helpful to others in the community.
Thank you.
Hi devopsfj,
I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to "Accept " the answer.
Issue:
Azure Kubernetes Service - Why Are System Pods Being Scheduled On User Node Pools?
Solution:
I believe I may have found the issue to what was happening...
I believe my System Nodes did not have the accepted amount of CPU/Memory left to facilitate all system pods with the CriticalAddonsOnly:true:NoSchedule taint.
Once I increased my system nodes to have the recommended specs as stated here on MS Docs.
All of the system pods were schedules on the system nodes (apart from Daemon Sets of course)
If you have any other questions or are still running into more issues, please let me know. Thank you again for your time and patience throughout this issue.
Please remember to "Accept Answer" if any answer/reply helped, so that others in the community facing similar issues can easily find the solution.
Thank you.
Hi devopsfj,
I hope you're doing well. Just checking in to see if you had a chance to see my response to your question. If you have any further queries, please do let us know.
If the answer is helpful, please click "Accept Answer" and "Upvote it" as it can be helpful to others in the community.