we have nodes without pmi which is impacting some jobs https://monasheresearch.freshdesk.com/a/tickets/28868 code tested on massive004. It only works on new slurm installs.
to fix the nodes already in the queue run
cd /opt/src/slurm-20.02.7/contribs/pmi;sudo make install
cd /opt/src/slurm-20.02.7/contribs/pmi2;sudo make install