Mellanox config

factoring out mellanox_config from mellanox_install

TODO:

Host - Configuration of MLNX NIC:

Note: Disable Global Pause on all NICs

Step 1 - Set QoS parameters
### Use all commands and setup a startup script to apply on boot
Set DSCP (L3) as trust mode for the NIC

• # mlnx_qos -i --trust dscp

Set ToS to 106 (DSCP 26) for ALL RoCE traffic (Note: This command is nonpersistent)

• # echo 106 > /sys/class/infiniband//tc/1/traffic_class

Set the RDMA-CM ToS to 106 (DSCP 26) (Note: This command is nonpersistent)

• # cma_roce_tos -d -t 106

Enable ECN for TCP traffic (Note: This command is nonpersistent)

• # sysctl -w net.ipv4.tcp_ecn=1

Step 2 - Enable PFC on RoCE prioritry
Activate PFC on priority 3
Using mlnx_qos tool (Note: This command is nonpersistent):

• # mlnx_qos -i --pfc 0,0,0,1,0,0,0,0

``` Tuning of MLNX NIC: Mellanox NIC tuning can be done by using the mlnx_tune tool included in the MLX OFED Drivers. It can also be downloaded separately if you are using the inbox drivers.

From discussion with Monash, we do not need to tune the cards as the default profile is working correctly.

Commands to run: mlnx_tune -r - Show a report of the current MLX NIC and System

- [x] # mlnx_tune -p <profile-name>  //There are several profiles to select from, see below. High Throughput: