Is it possible (sensible?) to run Docker containers on Azure CycleCloud using Slurm?

Gary Mansell 20

I have been successfully running Azure CycleCloud & Slurm scheduler for running our HPC (CFD & CAE) Analysis Solving jobs from a /shared/apps loadpoint in a regular manner.

I demo'd our HPC Solving capabilities to our Climate modelling team and they are super interested in moving to an Azure CycleCloud and Slurm environment too, if it were possible, as they can scale in/out with workload and run huge jobs in parallel, rather than take weeks on their current physical kit.

But, they run their simulations inside Docker containers - So, is it possible (or is it even sensible?) to run Docker containers in an Azure CycleCloud / Slurm environment? Can anyone advise me on the pros/cons of this?

If it is indeed possible and not non-sensical - perhaps someone could share some information on what tools/techniques that I might need and how to do it?

Thanks

Gary

1 answer

vipullag-MSFT 26,421 Reputation points

2024-04-15T17:46:57.36+00:00

Hello Gary Mansell

Welcome to Microsoft Q&A Platform, thanks for posting your query here.

I just checked with internal team on this ask. Yes, it is possible running some distributed workload through docker containers on ndv5 vmss using slurm+ containers.

Mostly with the pyxis and enroot tool from NVIDIA. Cyclecloud has support for those plugins and you can test with some simple mpi collectives from within a container across nodes.

Ref: https://github.com/NVIDIA/pyxis
Also, refer tto this Cyclecloud pyxis+enroot project https://github.com/Azure/azurehpc/tree/master/experimental/cc_slurm_pyxis_enroot

Hope this helps.
Please sign in to rate this answer.
Gary Mansell 20 Reputation points

2024-04-16T07:23:08.86+00:00

@vipullag-MSFT thanks for getting back to me, I did spot those repo's, but they maybe look a little experimental for Enterprise usage.

It seems to me that the main issue with running Docker containers directly on a HPC Cluster is that they need to run with root access to get access to the underlying hardware components (Infiniband & GPU etc.) in an HPC Cluster - is that correct?

There seems to be an application called Singularity that will run Docker containers and might work around this problem, but it's not clear to me how I go about downloading/converting the Docker containers to run them on a CycleCloud cluster with Slurm and Singlularity.

I also am not sure how things work with multiple nodes/processes/cores/threads and MPI when running a Docker container with Singularity using Slurm - so I wonder if anyone can explain how that might work (as our climate sim needs to run both single host and multi-host with many processes/threads).

Presumably, you only want one container per node and for multiple hosts - you need the container mounted for each host - does this just "magically happen" with Singularity or is there parameters which control this?

Also, how do you control how many processes/threads that are launched with Slurm across each of the nodes - do you just use the Slurm variables, or do things need to be set within the container too?

vipullag-MSFT 26,421 Reputation points

2024-04-17T04:18:23.1166667+00:00

Hello Gary Mansell

Thanks for responding back and sharing the details.

I checked with internal team on this, and can confirm that nvidia will work you just have to make sure that the users on the box have the same group as the group that owns the devices on the host.

For example if you ls -al /dev | grep nvidia you should see your various nvidia devices. those shoudl be owend by root, but part of a group (probably vglusers)

Find that group id cat /etc/groups | grep vglusersand then add a user to yoru container who is part of that group and the nonroot use shoudl have access to the devices.

Ref: https://techcommunity.microsoft.com/t5/azure-high-performance-computing/running-hpc-and-ai-workloads-in-containers-in-azure/ba-p/3941113
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Is it possible (sensible?) to run Docker containers on Azure CycleCloud using Slurm?

1 answer

Your answer