Traceback (most recent call last):
 File "/usr/local/sbin/provision_slurm.py", line 100, in <module>
   mk_slurmuser_batch(usergroup,"default")
 File "/usr/local/sbin/provision_slurm.py", line 67, in mk_slurmuser_batch
   userdict = defaultdict(list)
NameError: global name 'defaultdict' is not defined

Update provision_slurm.py.j2 to fix logic in mk_slurmuser_batch. This previously did not add any users if one user in the list was a member of 'default'

tempalte xorg.conf when running on a node with 1GPU (this template works for M3 K1 nodes, will need to do something smarter for other clusters with GPUS)

The GPU role used to just detect if any driver was installed and skip installation if one was found
it will now execute nvidia-smi to determine the current driver version and compare it against the desired
driver version, installing in the case of mismatch

Note that if an existing driver is installed, the nvidia persistence daemon must be stopped before installation can proceed

I had the syntax incorrect when the glsuter cluster consists of more than two nodes. We're not attempting to use three node clusters so we can't get split brain

this was initially removed as it resulted in programs in /etc/profile.d searching /usr/local/bin, which if the NFS server had hung resulted in ec2-user not being able to log in
subsequently verified that it seems safe to add it to the end of the path, obviously subject to not having anything called from profile that exists in /usr/local

add check_mode tags to shell command and script tasks that should be run even when checking (only do this if the task is known to not make system changes

add check_mode tags to shell command and script tasks that should be run even when checking (only do this if the task is known to not make system changes

Simple at this stage as N(Sockets)==2

renamed files so they are executed last in /etc/profile.d (Otherwise default module init routines overwrite changes)

1) provision_homedir was not correctly chowning directories
2) provision_homedir should skip existing directories for speed