6

I started to get "not enough free space" errors when trying to copy files, and at some point I had trouble booting and did a ZFS rollback. I still get space errors despite (according to df and du below) <60% drive space being used.

The answer below points to zfs snapshots taking up too much space, and it's a helpful start. But sudo du /.zfs/snapshot/ -h --max-depth=0 returns only 60G. Are there 140G of snapshots elsewhere? (or what else could it be?)

Also conspicuously zfs list -t snapshot doesn't show the (biggest and I assume most defunct?) dataset rpool/USERDATA/thomas_fv2wpz which is shown in zfs list output below to be 278G.

Here's a table of contents for outputs:

  • uname -a
  • zfs list
  • zfs list -o space rpool/USERDATA
  • Should I omit (some/parts of) the following long outputs? this my first time asking a question.
  • df -h
  • du -h --max-depth=1
  • lsblk -f
$ uname -a
Linux thomas-Latitude-E6530 5.4.0-31-generic #35-Ubuntu SMP Thu May 7 20:20:34 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

$ zfs list
NAME                                               USED  AVAIL     REFER  MOUNTPOINT
bpool                                              272M  1.48G       96K  /boot
bpool/BOOT                                         270M  1.48G       96K  none
bpool/BOOT/ubuntu_nhdx6z                           270M  1.48G      180M  /boot
bpool/BOOT/ubuntu_smztg7                             8K  1.48G      180M  /boot
rpool                                              441G  4.89G       96K  /
rpool/ROOT                                        8.97G  4.89G       96K  none
rpool/ROOT/ubuntu_nhdx6z                          8.51G  4.89G     3.69G  /
rpool/ROOT/ubuntu_nhdx6z/srv                       264K  4.89G       96K  /srv
rpool/ROOT/ubuntu_nhdx6z/usr                       872K  4.89G       96K  /usr
rpool/ROOT/ubuntu_nhdx6z/usr/local                 776K  4.89G      144K  /usr/local
rpool/ROOT/ubuntu_nhdx6z/var                      3.24G  4.89G       96K  /var
rpool/ROOT/ubuntu_nhdx6z/var/games                 152K  4.89G       96K  /var/games
rpool/ROOT/ubuntu_nhdx6z/var/lib                  3.10G  4.89G     2.74G  /var/lib
rpool/ROOT/ubuntu_nhdx6z/var/lib/AccountsService   688K  4.89G      104K  /var/lib/AccountsService
rpool/ROOT/ubuntu_nhdx6z/var/lib/NetworkManager   1.76M  4.89G      164K  /var/lib/NetworkManager
rpool/ROOT/ubuntu_nhdx6z/var/lib/apt              70.1M  4.89G     53.6M  /var/lib/apt
rpool/ROOT/ubuntu_nhdx6z/var/lib/dpkg             97.8M  4.89G     39.4M  /var/lib/dpkg
rpool/ROOT/ubuntu_nhdx6z/var/log                   138M  4.89G     36.4M  /var/log
rpool/ROOT/ubuntu_nhdx6z/var/mail                  152K  4.89G       96K  /var/mail
rpool/ROOT/ubuntu_nhdx6z/var/snap                 1.31M  4.89G      240K  /var/snap
rpool/ROOT/ubuntu_nhdx6z/var/spool                 848K  4.89G      120K  /var/spool
rpool/ROOT/ubuntu_nhdx6z/var/www                   152K  4.89G       96K  /var/www
rpool/ROOT/ubuntu_smztg7                           471M  4.89G     3.18G  /
rpool/ROOT/ubuntu_smztg7/srv                        56K  4.89G       96K  /srv
rpool/ROOT/ubuntu_smztg7/usr                       144K  4.89G       96K  /usr
rpool/ROOT/ubuntu_smztg7/usr/local                 144K  4.89G      144K  /usr/local
rpool/ROOT/ubuntu_smztg7/var                       203M  4.89G       96K  /var
rpool/ROOT/ubuntu_smztg7/var/games                 112K  4.89G       96K  /var/games
rpool/ROOT/ubuntu_smztg7/var/lib                   177M  4.89G     2.65G  /var/lib
rpool/ROOT/ubuntu_smztg7/var/lib/AccountsService   144K  4.89G      104K  /var/lib/AccountsService
rpool/ROOT/ubuntu_smztg7/var/lib/NetworkManager    320K  4.89G      148K  /var/lib/NetworkManager
rpool/ROOT/ubuntu_smztg7/var/lib/apt              4.08M  4.89G     53.3M  /var/lib/apt
rpool/ROOT/ubuntu_smztg7/var/lib/dpkg             14.0M  4.89G     39.4M  /var/lib/dpkg
rpool/ROOT/ubuntu_smztg7/var/log                  24.9M  4.89G     15.0M  /var/log
rpool/ROOT/ubuntu_smztg7/var/mail                  112K  4.89G       96K  /var/mail
rpool/ROOT/ubuntu_smztg7/var/snap                  212K  4.89G      188K  /var/snap
rpool/ROOT/ubuntu_smztg7/var/spool                 160K  4.89G      120K  /var/spool
rpool/ROOT/ubuntu_smztg7/var/www                   112K  4.89G       96K  /var/www
rpool/USERDATA                                     432G  4.89G       96K  /
rpool/USERDATA/helper_zthl10                       180K  4.89G      124K  /home/helper
rpool/USERDATA/root_fv2wpz                         316K  4.89G     1.12M  /root
rpool/USERDATA/root_mhe8ya                        1.78M  4.89G      992K  /root
rpool/USERDATA/thomas_fv2wpz                       971M  4.89G      278G  /home/thomas
rpool/USERDATA/thomas_mhe8ya                       431G  4.89G      217G  /home/thomas

$ zfs list -o space rpool/USERDATA
NAME            AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
rpool/USERDATA  4.88G   432G        0B     96K             0B       432G

$ df -h
Filesystem                                        Size  Used Avail Use% Mounted on
udev                                              5.8G     0  5.8G   0% /dev
tmpfs                                             1.2G  2.1M  1.2G   1% /run
rpool/ROOT/ubuntu_nhdx6z                          8.8G  3.7G  5.1G  42% /
tmpfs                                             5.8G  253M  5.6G   5% /dev/shm
tmpfs                                             5.0M  4.0K  5.0M   1% /run/lock
tmpfs                                             5.8G     0  5.8G   0% /sys/fs/cgroup
rpool/USERDATA/helper_zthl10                      5.1G  128K  5.1G   1% /home/helper
bpool/BOOT/ubuntu_nhdx6z                          1.7G  180M  1.5G  11% /boot
rpool/USERDATA/thomas_fv2wpz                      284G  279G  5.1G  99% /home/thomas
rpool/USERDATA/root_fv2wpz                        5.1G  1.0M  5.1G   1% /root
rpool/ROOT/ubuntu_nhdx6z/srv                      5.1G  128K  5.1G   1% /srv
rpool/ROOT/ubuntu_nhdx6z/var/games                5.1G  128K  5.1G   1% /var/games
rpool/ROOT/ubuntu_nhdx6z/var/log                  5.2G   29M  5.1G   1% /var/log
rpool/ROOT/ubuntu_nhdx6z/var/lib                  7.9G  2.8G  5.1G  35% /var/lib
rpool/ROOT/ubuntu_nhdx6z/usr/local                5.1G  256K  5.1G   1% /usr/local
rpool/ROOT/ubuntu_nhdx6z/var/mail                 5.1G  128K  5.1G   1% /var/mail
rpool/ROOT/ubuntu_nhdx6z/var/www                  5.1G  128K  5.1G   1% /var/www
/dev/sda1                                         511M  7.1M  504M   2% /boot/efi
rpool/ROOT/ubuntu_nhdx6z/var/snap                 5.1G  256K  5.1G   1% /var/snap
rpool/ROOT/ubuntu_nhdx6z/var/spool                5.1G  128K  5.1G   1% /var/spool
rpool/ROOT/ubuntu_nhdx6z/var/lib/AccountsService  5.1G  128K  5.1G   1% /var/lib/AccountsService
/dev/loop2                                         62M   62M     0 100% /snap/caprine/36
/dev/loop3                                         55M   55M     0 100% /snap/core18/1754
/dev/loop4                                        176M  176M     0 100% /snap/gimp/273
/dev/loop0                                        9.2M  9.2M     0 100% /snap/canonical-livepatch/95
/dev/loop5                                        241M  241M     0 100% /snap/gnome-3-34-1804/24
rpool/ROOT/ubuntu_nhdx6z/var/lib/dpkg             5.2G   40M  5.1G   1% /var/lib/dpkg
rpool/ROOT/ubuntu_nhdx6z/var/lib/apt              5.2G   54M  5.1G   2% /var/lib/apt
rpool/ROOT/ubuntu_nhdx6z/var/lib/NetworkManager   5.1G  256K  5.1G   1% /var/lib/NetworkManager
/dev/loop7                                        174M  174M     0 100% /snap/gimp/252
/dev/loop6                                         94M   94M     0 100% /snap/core/9066
/dev/loop1                                         55M   55M     0 100% /snap/core18/1705
/dev/loop8                                        164M  164M     0 100% /snap/spotify/41
/dev/loop9                                        256K  256K     0 100% /snap/gtk2-common-themes/9
/dev/loop10                                       143M  143M     0 100% /snap/slack/23
/dev/loop11                                       256M  256M     0 100% /snap/gnome-3-34-1804/33
/dev/loop13                                        63M   63M     0 100% /snap/gtk-common-themes/1506
/dev/loop12                                        28M   28M     0 100% /snap/snapd/7264
/dev/loop14                                        50M   50M     0 100% /snap/snap-store/454
/dev/loop15                                       142M  142M     0 100% /snap/zoom-client/83
/dev/loop16                                       161M  161M     0 100% /snap/gnome-3-28-1804/116
/dev/loop17                                        87M   87M     0 100% /snap/shotcut/64
/dev/loop18                                       291M  291M     0 100% /snap/vlc/1620
/dev/loop19                                       140M  140M     0 100% /snap/slack/24
/dev/loop20                                        50M   50M     0 100% /snap/snap-store/433
/dev/loop21                                       145M  145M     0 100% /snap/zoom-client/84
tmpfs                                             1.2G   52K  1.2G   1% /run/user/1000
/dev/loop22                                        64M   64M     0 100% /snap/caprine/37

$ du -h --max-depth=1
25K ./Music
1.0K    ./.ssh
1.0K    ./Templates
36K ./.gnome
30M ./.thunderbird
52M ./Documents
368M    ./snap
8.4M    ./.mozilla
203G    ./Pictures
302M    ./.config
1021M   ./.cache
1.0K    ./Public
9.5G    ./Downloads
101M    ./.local
7.0K    ./Steam
41G ./Videos
28K ./.gnupg
7.6G    ./Desktop
16G ./.steam
17K ./.pki
11K ./.psensor
279G    .


$ lsblk -f
NAME   FSTYPE     LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINT
loop0  squashfs                                                    0   100% /snap/canonical-livepatch/95
loop1  squashfs                                                    0   100% /snap/core18/1705
loop2  squashfs                                                    0   100% /snap/caprine/36
loop3  squashfs                                                    0   100% /snap/core18/1754
loop4  squashfs                                                    0   100% /snap/gimp/273
loop5  squashfs                                                    0   100% /snap/gnome-3-34-1804/24
loop6  squashfs                                                    0   100% /snap/core/9066
loop7  squashfs                                                    0   100% /snap/gimp/252
loop8  squashfs                                                    0   100% /snap/spotify/41
loop9  squashfs                                                    0   100% /snap/gtk2-common-themes/9
loop10 squashfs                                                    0   100% /snap/slack/23
loop11 squashfs                                                    0   100% /snap/gnome-3-34-1804/33
loop12 squashfs                                                    0   100% /snap/snapd/7264
loop13 squashfs                                                    0   100% /snap/gtk-common-themes/1506
loop14 squashfs                                                    0   100% /snap/snap-store/454
loop15 squashfs                                                    0   100% /snap/zoom-client/83
loop16 squashfs                                                    0   100% /snap/gnome-3-28-1804/116
loop17 squashfs                                                    0   100% /snap/shotcut/64
loop18 squashfs                                                    0   100% /snap/vlc/1620
loop19 squashfs                                                    0   100% /snap/slack/24
loop20 squashfs                                                    0   100% /snap/snap-store/433
loop21 squashfs                                                    0   100% /snap/zoom-client/84
loop22 squashfs                                                    0   100% /snap/caprine/37
sda                                                                         
├─sda1 vfat             105C-066C                             503.9M     1% /boot/efi
├─sda2                                                                      
├─sda5 swap             ca95a35a-589b-4127-91d0-ba74f05c7011                [SWAP]
├─sda6 zfs_member bpool 10888252588400514123                                
└─sda7 zfs_member rpool 8231086401312639965                                 
sr0
Thomas
  • 61
  • 1
  • 5
  • Please add the output of `df -h`, also maybe add `-h` to your `du` command ... `203G` is more readable than `212714827`, also your home directory is not the only one of interest..., depending on the programs you have installed there might be a lot of data in `/var` or `/tmp` – pLumo Jun 02 '20 at 18:48
  • Do you know `baobab`? Its a GUI Tool to analyse the data usage on the whole partition or on a folder. With `du --max-depth=1` you only check the current folder. An other nice script is `lsblk -f` that gives you an overwiev over all partitions. – Boba Fit Jun 02 '20 at 18:49
  • Run sudo gparted and make sure there's no exclamation mark shown by the root partition. If you haven't got gparted then install it. – Paul Benson Jun 02 '20 at 19:34
  • Thanks @pLumo. Question was updated with both of those suggestions. Also: $ sudo du -h --max-depth=1 for /var and /tmp show them to be 3.5G and 70K respectively (can post code if that's helpful). – Thomas Jun 02 '20 at 20:30
  • Thanks @BobaFit. I have used the Disk Usage Analyser. It thinks that / is only 9.4G total, and (similar to df) doesn't show the true space of the drive. not sure if it's relevant, but rpoo and bpool are both unmounted. – Thomas Jun 02 '20 at 20:34
  • Thanks @PaulBenson. gparted shows root to be exclamation point -less (pun?). – Thomas Jun 02 '20 at 20:44
  • @Thomas Did you check if gparted also showed the root partition as full (ie > 60% used)? – Paul Benson Jun 03 '20 at 13:10
  • @PaulBenson see gparted output. also added `zpool list`, which is telling and I'm investigating further... – Thomas Jun 03 '20 at 14:53
  • And the output of `zfs list` ? – pLumo Jun 03 '20 at 15:13
  • 1
    see also https://www.reddit.com/r/zfs/comments/4v8wc2/zfs_reporting_incorrect_available_space/ – pLumo Jun 03 '20 at 15:29
  • 1
    `zfs list -t snapshot` and `zfs list -o space rpool/USERDATA` – pLumo Jun 03 '20 at 15:52
  • @pLumo updated. – Thomas Jun 03 '20 at 21:00

2 Answers2

5

Ubuntu 20.04 introduced experimental ZFS support with zsys (ZFS SYStem integration) tool. It creates snapshots on most apt operations for that you will be able to rollback a previous state if something fails.

Ubuntu has a ZFS system tool called Zsys, which provides automated system and user state saving. It also integrates better with GRUB so a user can revert to an earlier system state before booting into the desktop. (via)

Downside is of course that these will also eat up your available space.

Use zfs list -t snapshot to list the snapshots:

$ zfs list -t snapshot
NAME                   USED  AVAIL  REFER  MOUNTPOINT
...
rpool/example@snap1       -      -      -  -
...

and zfs destroy to delete a snapshot, e.g.:

zfs destroy rpool/example@snap1

Read also the official tutorial.

There is also a similar question with not many answers yet on how to disable automatic snapshot creation and another one asking for some official documentation (I couldn't find anything either).

pLumo
  • 26,204
  • 2
  • 57
  • 87
  • Correct answer. The snapshots are hidden with the Ubuntu setup, which leads to this confusion. Had to learn this myself. – emk2203 Jun 03 '20 at 16:44
  • just destroyed all but a handful of snapshots. disk usage and free space look the same. – Thomas Jun 04 '20 at 17:46
  • This seems borderline dangerous... I only accidentally saw that snapshots weren't being made during an update from the GUI because I clicked the thingy to see what what going on, and even then luckily happened to be glancing at the output and managed to remember the error message long enough to search Google for it. If any of that had not happened I would still be in the dark about snapshots no longer working, and as it is, supposing the last update left my system in an unbootable state I would have to go back an additional snapshot to recover since it didn't just make one. – Michael Jul 28 '21 at 16:30
  • How remove only large snapshots? – Fakhamatia Apr 05 '22 at 15:43
2

You could also use this for loop I wrote to just wipe out all of a specific pool. I wiped out rpool and bpool as I take backups of my PC with something else daily anyway. Just be sure to change from bpool to rpool if you only want one or the other.

for i in $(sudo zfs list -t snapshot | grep bpool | awk '{print $1}'); do sudo zfs destroy -R $i;done
BeastOfCaerbannog
  • 12,964
  • 10
  • 49
  • 77
kamehamehotdogs
  • 21
  • 1