How to measure GPU usage?

Question

Using the top command in the terminal lists processes, sorted by their CPU usage (and you can change it to sort by another parameter)

Is there an equivalent for the GPU?

This fellow is asking about RAM used by GPU

Deppending, if you are using a radeon you can use [radeontop](http://askubuntu.com/a/382160/169736), for nvidia there's another tool but I don't have the name at hand. — Braiam, Dec 07 '13 at 19:39
For nvidia it's the commandline tool `nvidia-smi`, except for the jetsons, where it is `tegrastats`. — Jus, Feb 17 '20 at 07:49
For a nvidia gpu, you can use `nvidia-smi -l 5`, which will provide an update every 5 seconds. (Change this number to update at a different interval.) — mikey, Apr 13 '21 at 11:14

score 154 · Accepted Answer · edited May 04 '18 at 16:09

154

For Nvidia GPUs there is a tool nvidia-smi that can show memory usage, GPU utilization and temperature of GPU.
For Intel GPU's you can use the intel-gpu-tools.
AMD has two options
1. fglrx (closed source drivers):
```
aticonfig --odgc --odgt
```
2. And for mesa (open source drivers), you can use RadeonTop .

^{Source:GPU usage monitoring}

edited May 04 '18 at 16:09

You'reAGitForNotUsingGit

14,669
9
48
83

answered Dec 07 '13 at 19:39

Mitch

106,657
24
210
268

85

Use `watch nvidia-smi` for real-time updates. – Lenar Hoyt Nov 20 '14 at 22:00
1

aticonfig won't work over SSH. Claims it needs an X server running to work (there is one running). However, RadeonTop (`sudo apt-get radeontop`) **does** work with the fglrx (needs root). Hurrah! Sadly RadeonTop doesn't provide any temperature readings. – Ken Sharp Jan 05 '15 at 09:58
aticonfig WILL work over SSH, but an X server with tcp enabled needs to be running. This can be done by configuring lightdm via `xserver-allow-tcp=true`. Searching around this site with these keywords should lead to the result. – jyalim Feb 18 '16 at 00:20
2

Not sure why but `watch -n 1 nvidia-smi` gave me real-time updates. `watch nvidia-smi` has a 2 sec update delay. – markroxor Aug 06 '18 at 07:58
8

`sudo intel_gpu_top` should give you real time updates for intel gpus. – George D Aug 07 '18 at 12:07
I like the radeontop. – Melroy van den Berg Dec 21 '19 at 01:12
You can use `nvidia-smi -l 1` for real-time updates. Source: [GPU usage monitoring (CUDA)](https://unix.stackexchange.com/questions/38560/gpu-usage-monitoring-cuda#:~:text=For%20linux%2C%20use%20nvidia%2Dsmi,refresh%20interval%20of%201%20second.) – Vthechamp Jun 06 '21 at 14:35

score 50 · Answer 2 · answered Oct 01 '16 at 19:45

50

You can use gpustat, which is a simple command-line script (wrapper for nvidia-smi) for querying and monitoring GPU status:

answered Oct 01 '16 at 19:45

Franck Dernoncourt

3,390
7
37
55

10

NOTE: Only for Nvidia – Mr. Developerdude Apr 22 '19 at 22:32
7

`pip install gpustat` – Shital Shah Sep 17 '19 at 08:09
1

Works perfect with proprietary driver, thank you. – Smeterlink Apr 26 '21 at 12:02

score 46 · Answer 3 · edited Sep 02 '18 at 16:35

46

For Intel:

Install intel-gpu-tools (its likely that they are installed already)
```
sudo apt-get install intel-gpu-tools 
```
Start the top like utility with
```
sudo intel_gpu_top
```
Check your stats and then exit with Ctrl+C

Thats what you get:

Thanks @Mitch! :)

edited Sep 02 '18 at 16:35

abu_bua

10,473
10
45
62

answered Apr 22 '17 at 09:10

Thomas Venturini

709
7
11

1

Nope, it doesn't seem to be installed by default. I'm using Skylake GT2 (Intel HD Graphics 520). – wyphan Feb 24 '21 at 20:54

score 37 · Answer 4 · edited Apr 13 '17 at 12:37

Nvidia: to continuously update the output of nvidia-smi, you can use nvidia-smi --loop=1 (refresh interval of 1 second) or nvidia-smi --loop-ms=1000 (refresh interval of 1000 milliseconds).

   -l SEC, --loop=SEC
       Continuously  report  query data at the specified interval, rather than
       the default of  just  once.   The  application  will  sleep  in-between
       queries.   Note  that on Linux ECC error or XID error events will print
       out during the sleep period if the -x flag was not specified.  Pressing
       Ctrl+C at any time will abort the loop, which will otherwise run indef‐
       initely.  If no argument is specified for the -l form a default  inter‐
       val of 5 seconds is used.

   -lms ms, --loop-ms=ms
       Same as -l,--loop but in milliseconds.

FYI:

WinEunuuchs2Unix · Answer 5 · 2019-07-08T10:35:06.627

Conky

I like to use conky as a real-time monitor for both CPU and GPU. Installation is straightforward:

sudo apt install conky

Intel i7-6700HQ iGPU HD 530

In this instance I've booted using the integrated GPU rather than the nVidia GTX 970M:

The conky code adapts depending on if booted with prime-select intel or prime-select nvidia:

nVidia GPU GTX 970M

In this instance I've booted using the nVidia GTX 970M rather than the integrated GPU:

Conky code

The conky code was recently modified to auto-sense the GPU. Now it doesn't have to be hand modified when rebooting to a different GPU:

#------------+
# Intel iGPU |
#------------+
${color orange}${hr 1}${if_existing /sys/class/drm/card0/gt_cur_freq_mhz}
${color2}${voffset 5}Intel® Skylake GT2 HD 530 iGPU @${alignr}${color green}
${execpi .001 (cat /sys/class/drm/card0/gt_cur_freq_mhz)} MHz
${color}${goto 13}Min. Freq:${goto 120}${color green}${execpi .001 (cat /sys/class/drm/card0/gt_min_freq_mhz)} MHz${color}${goto 210}Max. Freq:${alignr}${color green}${execpi .001 (cat /sys/class/drm/card0/gt_max_freq_mhz)} MHz
${color orange}${hr 1}${else}
#------------+
# Nvidia GPU |
#------------+
#${color orange}${hr 1}${if_match "${lsmod | grep nvidia_uvm}">""}
${color2}${voffset 5}${execpi .001 (nvidia-smi --query-gpu=gpu_name --format=csv,noheader)} ${color1}@ ${color green}${execpi .001 (nvidia-smi --query-gpu=clocks.sm --format=csv,noheader)} ${alignr}${color1}Temp: ${color green}${execpi .001 (nvidia-smi --query-gpu=temperature.gpu --format=csv,noheader)}°C
${color1}${voffset 5}Ver: ${color green}${execpi .001 (nvidia-smi --query-gpu=driver_version --format=csv,noheader)} ${color1} P-State: ${color green}${execpi .001 (nvidia-smi --query-gpu=pstate --format=csv,noheader)} ${alignr}${color1}BIOS: ${color green}${execpi .001 (nvidia-smi --query-gpu=vbios_version --format=csv,noheader)}
${color1}${voffset 5}GPU:${color green}${execpi .001 (nvidia-smi --query-gpu=utilization.gpu --format=csv,noheader)} ${color1}Ram:${color green}${execpi .001 (nvidia-smi --query-gpu=utilization.memory --format=csv,noheader)} ${color1}Pwr:${color green}${execpi .001 (nvidia-smi --query-gpu=power.draw --format=csv,noheader)} ${alignr}${color1}Freq: ${color green}${execpi .001 (nvidia-smi --query-gpu=clocks.mem --format=csv,noheader)}
${color orange}${hr 1}${endif}

Different versions of the full code listing can be found in these answers:

Could you provide the steps to install conky and setup the theme to be like your Nvidia example please? — Tak, Jul 08 '19 at 08:14
@Tak I've updated answer with installation instructions and existing links to code. — WinEunuuchs2Unix, Jul 08 '19 at 10:36

Tshilidzi Mudau · Answer 6 · 2016-11-10T21:18:41.440

20

I use the following command:

nvidia-smi -l 2

and it gives me updates every 2 seconds.

Or :

watch -n0.1 "nvidia-settings -q GPUUtilization -q useddedicatedgpumemory"

And on AMD, use:

aticonfig --odgc --odgt

edited Nov 10 '16 at 21:18

answered Nov 10 '16 at 21:11

Tshilidzi Mudau

4,143
3
24
32

`useddedicatedgpumemory` is that the tensorcores utilization? – Fadwa Oct 28 '20 at 12:29
If you installed your nvidia driver, it's probably already installed. Take a look with `whereis nvidia-smi` to find its location. – bvdb Oct 30 '20 at 16:41

score 20 · Answer 7 · edited Apr 21 '23 at 10:26

20

You can use the monitoring program glances with its GPU monitoring plug-in:

open source
to install: sudo apt-get install -y python-pip; sudo pip install glances
to launch: sudo glances

It also monitors the CPU, disk IO, disk space, network, and a few other things:

edited Apr 21 '23 at 10:26

Lampe2020

51
9

answered Feb 07 '17 at 00:06

Franck Dernoncourt

3,390
7
37
55

You also need to do `pip install nvidia-ml-py3` – Paul Williams Aug 18 '18 at 04:40
1

It also works without `sudo` and when istalled for user only: `pip install --user glances[gpu]` and then run `glances`. – Ivan Kovtun May 10 '19 at 22:10

score 10 · Answer 8 · answered May 28 '20 at 19:42

10

I just found this command:

nvidia-smi --query-gpu=utilization.gpu --format=csv --loop=1

Here is a demo:

answered May 28 '20 at 19:42

singrium

6,532
7
38
68

score 9 · Answer 9 · answered Mar 03 '23 at 15:55

No one mentioned the nvtop ? A nice tool similar to the htop we used to. Installation is easy, simply do:

$ sudo apt install nvtop
or
$ snap install nvtop

The version from snap is usually a lot newer. The attractive feature is that it shows which process uses GPU by how much. A typical screenshot looks like the following:

score 5 · Answer 10 · answered Feb 24 '21 at 20:43

In my case nvidia-smi did not show the GPU load %, only the memory (guess my GTX 650 is too old).

What did work for me was the NVIDIA X Server Settings GUI app (shipped with the driver I believe). Navigate to the section named GPU 0 - (Your Model) - it shows the detailed status info of your GPU usage, updating every 2 seconds:

How to measure GPU usage?

10 Answers10

Conky

Intel i7-6700HQ iGPU HD 530

nVidia GPU GTX 970M

Conky code

Linked

Related