I have created a SLURM cluster on Azure CycleCloud and enabled autoscaling with a maximum of 20 nodes for HPC. I have verified that there is enough quota for at least 10 HPC and 2 HTC nodes. However, upon booting the cluster, only 5 nodes are available in HPC, and the number does not increase beyond that. I have tried several measures, including updating the configuration file to adjust the number of HPC nodes, using scontrol reconfigure , and running azslurm scale with root privileges, but none have been successful in increasing the total number of nodes. If anyone has insights or solutions to resolve this issue, it would be appreciated. Let me know if you need more info on this. Here is the output of the sinfo command for reference: PARTITION AVAIL TIMELIMIT NODES STATE NODELIST dynamic up infinite 0 n/a hpc* up infinite 5 idle~ ccw-hpc-[1-5] htc up infinite 2 idle~ ccw-htc-[1-2]

Hello @Parham Pourbozorgi Thanks for sharing the details. It appears that you’re using a customized cluster creation UI form (the default Slurm cluster creation form offers limits in terms of Cores rather than “Nodes”). Since the GUI is from a custom cluster template, it’s hard to know for sure what it is showing. But I’d guess that “Max HPC Nodes” maps to the MaxCount cluster parameter for the HPC Nodearray. In CycleCloud, both MaxCount and MaxCoreCount are user imposed limits on the autoscaler and do not reflect your Azure quota (so you could set have a lower quota than the limit set in Max HPC Nodes). An easy way to tell what your remaining Available Quota is for your cluster is to go to the Clusters page and click on the “Actions -> Add” action in the nodes table. That will show you the remaining cores available for use by the cluster. Here’s an example from one of my CycleCloud installs. It shows that I can create 39 more F2s_v2 VMs based on my current available Regional Quota of 100 cores (I must be using the other 22 cores for some other cluster). Note that there are 2 quotas (Regional + per-Family quota) PLUS my current MaxCoreCount limit that are limiting the number of nodes I can create. This is the number that determines the scale of the Slurm cluster. Can you check what your CycleCloud shows here? Tag me in your comments with the required details please.

Issue with Autoscaling Nodes in Azure CycleCloud SLURM Cluster

Accepted answer

Prrudram-MSFT 27,251 Reputation points

2024-11-10T17:02:39.6833333+00:00

Hello @Parham Pourbozorgi

Thanks for sharing the details. It appears that you’re using a customized cluster creation UI form (the default Slurm cluster creation form offers limits in terms of Cores rather than “Nodes”).

Since the GUI is from a custom cluster template, it’s hard to know for sure what it is showing. But I’d guess that “Max HPC Nodes” maps to the MaxCount cluster parameter for the HPC Nodearray. In CycleCloud, both MaxCount and MaxCoreCount are user imposed limits on the autoscaler and do not reflect your Azure quota (so you could set have a lower quota than the limit set in Max HPC Nodes).

An easy way to tell what your remaining Available Quota is for your cluster is to go to the Clusters page and click on the “Actions -> Add” action in the nodes table. That will show you the remaining cores available for use by the cluster.

Here’s an example from one of my CycleCloud installs. It shows that I can create 39 more F2s_v2 VMs based on my current available Regional Quota of 100 cores (I must be using the other 22 cores for some other cluster). Note that there are 2 quotas (Regional + per-Family quota) PLUS my current MaxCoreCount limit that are limiting the number of nodes I can create.

This is the number that determines the scale of the Slurm cluster.

Can you check what your CycleCloud shows here?

Tag me in your comments with the required details please.
Please sign in to rate this answer.

1 person found this answer helpful.
Parham Pourbozorgi 20 Reputation points

2024-11-11T06:58:21.89+00:00

Hi @Prrudram-MSFT ,

Thank you for the detailed information. I am not following the slurm creation format. I basically followed the slurm creation procedure from azure and is almost where I ended up following the GUI steps. The screenshot in previous post is accessible from Edit >> required settings (Image 1).

Nevertheless, I had previously checked these and made sure that the cluster can see my quotas. Below is a screenshot for adding a computing node (Image 2). As you can see it says that at least 7 more nodes are available for that type of machine but my cluster only sees 5 of them (Image 3). Not mentioning that I have increased the quota so much to be able to create even more than 7 nodes.

Please let me know if this information is clear enough and what can I do to be able to run more that 5 nodes on my cluster. If you require any other information please let me know so that I post them.

Much appreciated!

Prrudram-MSFT 27,251 Reputation points

2024-11-12T08:06:56.9733333+00:00

Hello @Parham Pourbozorgi

Thanks for getting back. I would recommend setting the Max HTC and Max HPC properties very high for now.
They are just the limits that take effect if you do not want Slurm to use your entire quota. Since you likely prefer to have Slurm use your full quota, I would set them to an outrageously high value (like 1000000) for now. (With a lower value, they’re just making debugging a bit more confusing.) I am discussing this with my azure cycle cloud product team internally, I will keep you posted once i have more details.

Parham Pourbozorgi 20 Reputation points

2024-11-12T08:38:08.2133333+00:00

Hello @Prrudram-MSFT

Thank you for the tip. I have set both HPC and HTC Max nodes to 1000000 now. As before it only increases my HTC nodes when I check. It does not change the HPC nodes. I must say that I have 32 CPU machines set for HTC nodes, that is why I only get 6 for that. HPC are 16 CPU machines but still they are not affected.

Prrudram-MSFT 27,251 Reputation points

2024-11-12T09:38:49.2166667+00:00

Hello @Parham Pourbozorgi

We have noticed that you have rated an answer as not helpful. We appreciate your feedback and are committed to improving your experience with the Q&A. I am working on finding solution to address your issue with product team. Please hold on while I get back to you with an answer.

Prrudram-MSFT 27,251 Reputation points

2024-11-13T18:32:41.8833333+00:00

Hello @Parham Pourbozorgi

Can you run azslurm limits and send the output of that command? Also, make sure that "Max VMs per VMSS" is at least 100 in the UI. For HPC clusters, we only create a single VMSS, so it can effectively limit the size. For the HTC cluster, we can create more than one, so this setting has no effect.

Parham Pourbozorgi 20 Reputation points

2024-11-14T06:39:42.1866667+00:00

Hi @Prrudram-MSFT

Indeed this was the solution. I increased the "Max VMs per VMSS" and now I can generate more HPC clusters. Thank you for patiently finding the solution.

Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

Prrudram-MSFT 27,251 Reputation points

2024-11-14T08:45:50.91+00:00

Hello Parham Pourbozorgi

I am glad to know the issue is fixed. Could you please be so kind to "Accept Answer" and Upvote at "Was it helpful"
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Issue with Autoscaling Nodes in Azure CycleCloud SLURM Cluster

0 additional answers

Your answer