This script enhances the functionality of nvidia-smi
and provides the following information:
Username
full process Command
GPU ID
PID
This is useful on multi-user servers and can be used to quickly identify which user is using the GPU and running what kind of program.
nvidia-smi && (nvidia-smi |tr -s ' '|grep -Eo "| [0123456789]+ N/A N/A [0-9]{3,} .*"|awk -F' ' '{system("s=$(cat /proc/"$4"/cmdline| tr \"\\0\" \" \");u=$(ps -o uname= -p "$4");echo "$1"sep"$4"sep$u sep"$7"sep$s" ) }'|sed 's/sep/\t/g')```
The output is like this
+-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | 0 3296 C python 5825MiB | | 0 21266 C /opt/conda/bin/python 6319MiB | | 1 3296 C python 137MiB | | 1 8063 C /usr/bin/python 273MiB | | 1 21266 C /opt/conda/bin/python 4735MiB | | 1 30439 C /usr/bin/python 199MiB | | 2 3296 C python 137MiB | | 2 21266 C /opt/conda/bin/python 4735MiB | +-----------------------------------------------------------------------------+ 0 3296 user1 5825MiB python playground2.py 0 21266 user2 6319MiB /opt/conda/bin/python -u /cail/model.py 1 3299 root 137MiB python playground2.py 1 8063 root 273MiB /usr/bin/python -u /sentsim/cli_predict.py 1 21266 root 4735MiB /opt/conda/bin/python -u /cail/model.py 1 30439 root 199MiB /usr/bin/python -u /sentsim/train.py 2 3296 root 137MiB python playground2.py 2 21266 root 4735MiB /opt/conda/bin/python -u /cail/model.py
For convenience, you can alias the command.
That's extremely helpful, thanks