Node count in AKS increased without approval/manual operation after SP renewal

Question

Node count in AKS increased without approval/manual operation after SP renewal

Sajin Seby 0

After SP renewal, node scaled up in agent pools even when the condition for scaling was manual

Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Sajin Seby 0 Reputation points

2025-02-27T17:03:08.7533333+00:00

Hi Geethasri,

Thanks for the update. I have verified that the autoscaling is false by default and the scale method within the node pools is in the manual state even after the SP renewal. Also verified that no automation is configured wrt SP renewal. This is the 2nd time I've faced the issue this month!

Thank You.
Geethasri.V 15 Reputation points Microsoft External Staff

2025-02-28T05:43:14.4233333+00:00

Hi Sajin Seby,

It seems like something out of the ordinary initiated the scaling, despite autoscaling and manual scaling being properly configured. It might be that the SP renewal somehow triggered a reset or conflict in AKS configurations, resulting in unexpected behavior.

It might also be caused by an unseen dependency or default behavior in Azure following the SP renewal, even if no automation is implemented on your side. Since this happened twice this month, I’d recommend closely monitoring the AKS activity logs around the renewal time to see if any other changes or system-level actions are occurring that could be influencing scaling.

If you have any further queries, please let us know we are glad to help you.
Sajin Seby 0 Reputation points

2025-02-28T05:54:46.8466667+00:00

Hi Geethasri,

I could not find the option to raise a ticket. Every time I try to create a support ticket, it takes me to either documentation or Q&A page.
Geethasri.V 15 Reputation points Microsoft External Staff

2025-02-28T17:13:06.5733333+00:00

Hi Sajin Seby,

Please go through this document for your reference
https://learn.microsoft.com/en-us/azure/aks/scale-cluster?source=recommendations&tabs=azure-cli
If you have any further question let me know.

Thank you.
Geethasri.V 15 Reputation points Microsoft External Staff

2025-03-06T08:47:13.97+00:00

Hi Sajin Seby,

Just checking in to see if you have got a chance to see my response to your question in resolving the issue.

If it was helpful, please click "Upvote" on this post to let us know.

Thank You.
Sajin Seby 0 Reputation points

2025-03-07T04:03:16.7233333+00:00

Hi Geethasri,

Nothing helped me to resolve my issue. Also, I cannot raise a support request via Azure even though we have support plans. Every time it redirects me to the documentation page.

Thanks
Geethasri.V 15 Reputation points Microsoft External Staff

2025-03-07T12:45:52.8633333+00:00
Hi Sajin Seby,

Reasons for Node Count Increase

Cluster Autoscaler:

Automatically adjusts the number of nodes in response to resource demands.

If workloads increase, the autoscaler may add nodes without manual intervention.

Scheduled Jobs :

If there are scheduled tasks that require more resources, they could trigger an increase in node count.

Configuration Changes:

Changes in configuration or policies related to scaling might have been applied, leading to an increase in nodes.

Service Principal Permissions:

The renewed service principal may have permissions that allow it to modify the cluster settings, including scaling operations.

Monitoring and Alerts:

Check if there are any monitoring tools or alerts that might have triggered scaling actions based on predefined thresholds.

Recommended Actions

Review Cluster Autoscaler Settings:

Verify if the autoscaler is enabled and check its configuration.

Audit Logs:

Examine the audit logs for any actions taken by the service principal or other users that could explain the increase.

Scaling Policies:

Review any scaling policies that may have been set up to ensure they align with your operational requirements.

Service Principal Permissions:

Ensure that the service principal has the appropriate permissions and that no unnecessary permissions are granted. If you have any further queries, please let us know we are glad to help you. Thank you
Geethasri.V 15 Reputation points Microsoft External Staff

2025-03-10T08:48:51.6166667+00:00

Hi Sajin Seby,

I wanted to check if you had the opportunity to review the information which was provided in my previous posted comment.

If it was helpful, please click "Upvote" on this post to let us know.

Thank You.
Geethasri.V 15 Reputation points Microsoft External Staff

2025-03-10T15:54:04.7433333+00:00

Hi Sajin Seby,

I wanted to check if you had the opportunity to review the information which was provided in my previous posted comment.

If it was helpful, please click "Upvote" on this post to let us know.

Thank You.
Sajin Seby 0 Reputation points

2025-03-11T04:13:51.41+00:00

Hi,

Nothing seems to be helpful.

Thanks
Geethasri.V 15 Reputation points Microsoft External Staff

2025-03-11T10:57:35.9+00:00
Hi Sajin Seby,

Check Cluster Autoscaler Settings

Ensure that the cluster autoscaler is configured correctly.

Review the 'ENABLE_CLUSTER_AUTOSCALER 'parameter in your configuration file. If it's set to "true," the autoscaler will automatically adjust the number of nodes based on workload demands to your AKS workload.

Disable Autoscaler (if necessary)

If you enable autoscaler , just consider disabling the cluster autoscaler by setting 'ENABLE_CLUSTER_AUTOSCALER 'to "false."

This will prevent automatic adjustments to the node count, allowing you to control scaling through manual operations only.

Check the AKS cluster settings to ensure the scaling configuration is indeed set to manual.

Review any logs and events around the time of SP renewal to see if something triggered the scaling action.

Ensure that the Service Principal permissions are correctly configured post-renewal, especially for scaling-related operations.

Double-check if any automated scripts or Azure policies are in place that could have affected scaling behavior. If you have any further queries, please let us know we are glad to help you.
If it was helpful, please click "Upvote" on this post to let us know. Thank You.
Sajin Seby 0 Reputation points

2025-03-11T12:02:22.6266667+00:00

Hi,

I have verified the above but could find nothing useful.

Thanks

1 answer

Your answer

Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Sajin Seby 0 Reputation points

2025-02-27T17:03:08.7533333+00:00

Hi Geethasri,

Thanks for the update. I have verified that the autoscaling is false by default and the scale method within the node pools is in the manual state even after the SP renewal. Also verified that no automation is configured wrt SP renewal. This is the 2nd time I've faced the issue this month!

Thank You.
Geethasri.V 15 Reputation points Microsoft External Staff

2025-02-28T05:43:14.4233333+00:00

Hi Sajin Seby,

It seems like something out of the ordinary initiated the scaling, despite autoscaling and manual scaling being properly configured. It might be that the SP renewal somehow triggered a reset or conflict in AKS configurations, resulting in unexpected behavior.

It might also be caused by an unseen dependency or default behavior in Azure following the SP renewal, even if no automation is implemented on your side. Since this happened twice this month, I’d recommend closely monitoring the AKS activity logs around the renewal time to see if any other changes or system-level actions are occurring that could be influencing scaling.

If you have any further queries, please let us know we are glad to help you.
Sajin Seby 0 Reputation points

2025-02-28T05:54:46.8466667+00:00

Hi Geethasri,

I could not find the option to raise a ticket. Every time I try to create a support ticket, it takes me to either documentation or Q&A page.
Geethasri.V 15 Reputation points Microsoft External Staff

2025-02-28T17:13:06.5733333+00:00

Hi Sajin Seby,

Please go through this document for your reference
https://learn.microsoft.com/en-us/azure/aks/scale-cluster?source=recommendations&tabs=azure-cli
If you have any further question let me know.

Thank you.
Geethasri.V 15 Reputation points Microsoft External Staff

2025-03-06T08:47:13.97+00:00

Hi Sajin Seby,

Just checking in to see if you have got a chance to see my response to your question in resolving the issue.

If it was helpful, please click "Upvote" on this post to let us know.

Thank You.
Sajin Seby 0 Reputation points

2025-03-07T04:03:16.7233333+00:00

Hi Geethasri,

Nothing helped me to resolve my issue. Also, I cannot raise a support request via Azure even though we have support plans. Every time it redirects me to the documentation page.

Thanks
Geethasri.V 15 Reputation points Microsoft External Staff

2025-03-07T12:45:52.8633333+00:00

Hi Sajin Seby,

Reasons for Node Count Increase

Cluster Autoscaler:

Automatically adjusts the number of nodes in response to resource demands.

If workloads increase, the autoscaler may add nodes without manual intervention.

Scheduled Jobs :

If there are scheduled tasks that require more resources, they could trigger an increase in node count.

Configuration Changes:

Changes in configuration or policies related to scaling might have been applied, leading to an increase in nodes.

Service Principal Permissions:

The renewed service principal may have permissions that allow it to modify the cluster settings, including scaling operations.

Monitoring and Alerts:

Check if there are any monitoring tools or alerts that might have triggered scaling actions based on predefined thresholds.

Recommended Actions

Review Cluster Autoscaler Settings:

Verify if the autoscaler is enabled and check its configuration.

Audit Logs:

Examine the audit logs for any actions taken by the service principal or other users that could explain the increase.

Scaling Policies:

Review any scaling policies that may have been set up to ensure they align with your operational requirements.

Service Principal Permissions:

Ensure that the service principal has the appropriate permissions and that no unnecessary permissions are granted. If you have any further queries, please let us know we are glad to help you. Thank you
Geethasri.V 15 Reputation points Microsoft External Staff

2025-03-10T08:48:51.6166667+00:00

Hi Sajin Seby,

I wanted to check if you had the opportunity to review the information which was provided in my previous posted comment.

If it was helpful, please click "Upvote" on this post to let us know.

Thank You.
Geethasri.V 15 Reputation points Microsoft External Staff

2025-03-10T15:54:04.7433333+00:00

Hi Sajin Seby,

I wanted to check if you had the opportunity to review the information which was provided in my previous posted comment.

If it was helpful, please click "Upvote" on this post to let us know.

Thank You.
Sajin Seby 0 Reputation points

2025-03-11T04:13:51.41+00:00

Hi,

Nothing seems to be helpful.

Thanks
Geethasri.V 15 Reputation points Microsoft External Staff

2025-03-11T10:57:35.9+00:00

Hi Sajin Seby,

Check Cluster Autoscaler Settings

Ensure that the cluster autoscaler is configured correctly.

Review the 'ENABLE_CLUSTER_AUTOSCALER 'parameter in your configuration file. If it's set to "true," the autoscaler will automatically adjust the number of nodes based on workload demands to your AKS workload.

Disable Autoscaler (if necessary)

If you enable autoscaler , just consider disabling the cluster autoscaler by setting 'ENABLE_CLUSTER_AUTOSCALER 'to "false."

This will prevent automatic adjustments to the node count, allowing you to control scaling through manual operations only.

Check the AKS cluster settings to ensure the scaling configuration is indeed set to manual.

Review any logs and events around the time of SP renewal to see if something triggered the scaling action.

Ensure that the Service Principal permissions are correctly configured post-renewal, especially for scaling-related operations.

Double-check if any automated scripts or Azure policies are in place that could have affected scaling behavior. If you have any further queries, please let us know we are glad to help you.
If it was helpful, please click "Upvote" on this post to let us know. Thank You.
Sajin Seby 0 Reputation points

2025-03-11T12:02:22.6266667+00:00

Hi,

I have verified the above but could find nothing useful.

Thanks

Answer 1

Hi Sajin Seby,

Welcome to the Microsoft Q&A Platform! Thank you for asking your question here.

We understand from your query that you are experiencing an issue with node count increases when you renewed SP.

It appears that the growth in node numbers may be because of automated functionality within AKS, such as the Cluster Auto scaler, which may have been activated following the SP renewal. When the Service Principal was renewed, it may have reinstated permissions or activated a policy to enable autoscaling to take place, particularly if there were resource requirements.

It's also conceivable that certain automated tasks or scripts involving the SP renewal inadvertently caused scaling activities. To avoid this in the future, I'd recommend verifying the autoscaling configuration and checking any automation associated with the SP to make sure everything is set up as desired.

If the answer was helpful, please don't forget to upvote and accept answer the answer

Thank you.

Share via

Node count in AKS increased without approval/manual operation after SP renewal

1 answer

Your answer