Simple at this stage as N(Sockets)==2

set the LimitMEMLOCK=infinity (whcih means jobs are unrestricted) hopefully this allows the use the mellanox communciation lib in mpi jobs

install gres.conf to the correct place. Hardcode the CPUs associated with the GPUs. This is a temprorary fix at best

add role ssh-keepalive, which uses  keepalive packets to prevent firewalls closing inactive ssh connections

depending upon node. This is done by a script nvidia-probe.py which probes node for correct information

add role enable_modules (and remove enable_lmod) use enable_modules to select either lmod or modulecmd being source from /etc/profile.d on Login and Compute nodes

allow provision_slurm to take a path to the lockfile so its on a directory shared between managment nodes