disk is full, but cannot find big files or folders

Question

Ubuntu server shows me that I use almost all disk:

Usage of /:   95.5% of 118.12GB

And I try to find big folders and files, run ncdu:

ncdu 1.8 ~ Use the arrow keys to navigate, press ? for help                                                                                                                                                 
--- / ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    5.5GiB [##########] /root                                                                                                                                                                               
    2.3GiB [####      ] /var
  628.6MiB [#         ] /usr
  209.9MiB [          ] /lib
   28.2MiB [          ] /boot
    8.6MiB [          ] /bin
    7.7MiB [          ] /sbin
    6.6MiB [          ] /etc
  208.0KiB [          ] /run
  112.0KiB [          ] /tmp
   48.0KiB [          ] /opt
e  16.0KiB [          ] /lost+found
    8.0KiB [          ] /dev
    8.0KiB [          ] /media
    4.0KiB [          ] /lib64
e   4.0KiB [          ] /srv
e   4.0KiB [          ] /selinux
e   4.0KiB [          ] /mnt
e   4.0KiB [          ] /home
    0.0  B [          ] /proc
    0.0  B [          ] /sys
@   0.0  B [          ]  initrd.img
@   0.0  B [          ]  vmlinuz

According to ncdu I use about 10 GiB of 128 GiB - it is about 10 %. Contradiction.

How to clean my ubutntu server without rebooting?

I thought that ncdu lies and used another apps to find big files and folders. All of them shows the same result as ncdu.

And df -h command shows that disk is full.

# df -h

Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda       119G  113G     0 100% /
udev            2.0G  8.0K  2.0G   1% /dev
tmpfs           788M  212K  788M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            2.0G     0  2.0G   0% /run/shm

Update

sudo du -sch /* result:

$ sudo du -sch

8.7M    /bin
29M /boot
8.0K    /dev
6.6M    /etc
4.0K    /home
0   /initrd.img
210M    /lib
4.0K    /lib64
16K /lost+found
8.0K    /media
4.0K    /mnt
48K /opt
du: cannot access `/proc/4470/task/4470/fd/4': No such file or directory
du: cannot access `/proc/4470/task/4470/fdinfo/4': No such file or directory
du: cannot access `/proc/4470/fd/4': No such file or directory
du: cannot access `/proc/4470/fdinfo/4': No such file or directory
0   /proc
5.0G    /root
212K    /run
7.8M    /sbin
4.0K    /selinux
4.0K    /srv
0   /sys
112K    /tmp
629M    /usr
2.3G    /var
0   /vmlinuz
8.1G    total

8.1G total as usual. But I see cannot access rows, maybe problem because of them.

Then I checked the biggest folder in /. It is /root :

$ sudo du -sch /root

96K /root/Downloads
2.5G    /root/Dropbox
36K /root/nohup.out
4.0K    /root/npm-debug.log
4.0K    /root/readonly
980K    /root/redis-2.6.16.tar.gz
228M    /root/tmp
2.7G    total

Just a thought, might check the contents of /var/log/ to see if any logs have grown expotenially. — Mordoc, Jan 31 '14 at 04:50
Try `du -sch /*` to see which root directories are using the most space, and descend from there into the places using the most space. — DopeGhoti, Jan 31 '14 at 05:21
@DopeGhoti I tried but saw the same about `8.1 GiB` full (added this to update). Cannot figure out where is the rest about `100 GiB` — Maxim Yefremov, Jan 31 '14 at 05:36
Do you have installed some game with Valve Steam program ? In case of 10 GB download then further 50 GB can be swallowed or more ... — dschinn1001, Jan 31 '14 at 07:18
@douggro rebooting helped, thanx. It's a pity that it is the only option. — Maxim Yefremov, Feb 04 '14 at 06:23

score 20 · Answer 1 · edited Apr 13 '17 at 12:24

20

I was running into this same issue on our lab machines and using this command

du -sch .[!.]* * |sort -h

I was able to find hidden files like inside of users trash bins that they had yet to delete.

Credit to here where I originally found this answer.

edited Apr 13 '17 at 12:24

Community

1

answered Feb 09 '16 at 09:33

Matt G.

406
3
7

Amazing solution! – AivanF. Nov 04 '18 at 15:08

score 16 · Answer 2 · edited Apr 04 '22 at 07:07

16

Check for deleted files that are still being held open by a process:

sudo lsof | grep deleted | less

That'll show the pid and file descriptor. I had this exact problem on a server, nothing in ncdu but disk filling up. It turned out to be a nightly process that moved files to a mounted samba share and occasionally didn't close out the file handle correctly, it seems.

If you find deleted files and want to clean them up, a reboot is probably easiest if that's acceptable. Or you can try killing the process. Or if you're sure they're not being used, you can manually empty them out, with something like this:
> /proc/14487/fd/12

edited Apr 04 '22 at 07:07

Pablo Bianchi

14,308
4
74
117

answered Jan 28 '17 at 14:21

Alric

261
2
3

This was my issue. Tomcat was holding an 80GB deleted file. A restart was enough to fix it. – AFP_555 Sep 23 '19 at 13:41
1

How can I delete them if the "reboot" command it's not enough? – sparkle Oct 15 '19 at 11:52
1

This shows many files what would I do with this info? I already rebooted and it didn't do anything. – Kvothe Sep 15 '22 at 15:13

score 4 · Answer 3 · edited Apr 04 '22 at 07:11

4

Following command will show disk utilization for /home directory with --max-depth=1:

sudo du -h -d 1 /

From manual:

-d, --max-depth=N print the total for a directory (or file, with --all) only if it is N or fewer levels below the command line argument; --max-depth=0 is the same as --summarize.

edited Apr 04 '22 at 07:11

Pablo Bianchi

14,308
4
74
117

answered Feb 23 '16 at 06:56

Kamchybek Jusupov

76
3

score 4 · Answer 4 · answered Nov 30 '17 at 11:13

4

We had this same issue and it turned out to be docker images, stored under var/lib/docker

ncdu does not list these as they are not visible to users. even running ncdu under sudo does not help.

This command purges all existing docker images...

docker rmi $(docker images -a -q)

answered Nov 30 '17 at 11:13

Baldy

141
3

1

Same issue here. In fact, even `docker system prune` wasn't finding everything. This command (which predates docker system prune) does the trick. – jscharf Dec 17 '18 at 22:13
2

recently we have discovered that `docker system prune -a -f` is a lot more thorough – Baldy Dec 20 '18 at 11:52

score 3 · Answer 5 · answered Jun 26 '20 at 08:17

It's possible that a process has opened a large file which has since been deleted. You'll have to kill that process to free up the space. You may be able to identify the process by using lsof. On Linux deleted yet open files are known to lsof and marked as (deleted) in lsof's output.

You can check this with sudo lsof +L1

please refer: https://serverfault.com/questions/315181/df-says-disk-is-full-but-it-is-not

score 3 · Answer 6 · answered May 01 '17 at 15:14

3

Be sure to check your disk mounts. None of the solutions I've seen here can identify space being taken up by a folder which has a mount placed over it.

answered May 01 '17 at 15:14

Rich Remer

313
3
11

any suggestion?i think this might be my problem – Eliethesaiyan Jan 23 '18 at 05:02
1

https://unix.stackexchange.com/questions/198542/what-happens-when-you-mount-over-an-existing-folder-with-contents – Rich Remer Jan 25 '18 at 15:59
Basically, check your existing mounts with `mount`, then add a second mount for each of the directories which have mounts placed over them. Then you can use normal disk tools like `du` on the newly created mount to see if it's the culprit. – Rich Remer Jan 25 '18 at 16:01

score 1 · Answer 7 · answered Nov 08 '21 at 20:56

One reason for "hidden" big files is a possibility to mount a folder (to some other folder/resource residing on a different disk) - the whole content of the original folder is kind of "hidden" - "du" won't show it, because it will be "overlapped" with the mounted resource.

Z-Bert · Answer 8 · 2022-12-12T18:15:14.597

I was experiencing this issue myself. Disk full, completely unable to locate the offending directories and/or files. However, I did solve this puzzle.

All of the previous answers are extremely helpful and legitimate but if you are, like me, unable to find the offending directories and/or files using the tips above, I'd like to share my solution.

I have an external USB drive (ex: dev/sdeX) that I used exclusively for backing up the content of root (ex: /dev/sdaX). Using that as a clue, I unmounted all mount points, excluding root and other required mount points. From there, I noticed the mount point to which the external USB drive mounts to is utilizing over 150G of space. Sure enough, I venture to this directory and see that it contains the same exact file structure and content as the external USB drive (when mounted). I mounted my external USB drive to verify backup data is intact (it is, check) then unmounted it and deleted the contents of that directory (for peace of mind, make sure to verify the mount point has been unmounted). Once I did that, usage on / (/dev/sdaX) went from 100% usage to 13%, which is right about what I expect.

Point of my answer... try unmounting all external drives and double check the mount point for data that should be on the external drive, not your root drive (/dev/sdaX).

Hope this helps.

score 0 · Answer 9 · edited Apr 04 '22 at 06:43

0

You can run the command below to find the top 10 files by disk usage:

find / -type f -printf '%s %p\n' 2>&1 \
     | grep -v 'Permission denied' \
     | sort -nr \
     | head -10

edited Apr 04 '22 at 06:43

Pedro

113
5

answered Aug 25 '19 at 09:58

ema

101
1

`find / -type f -printf '%s %p\n' 2>&1 \ | grep -v 'Permission denied' \ | sort -nr \ | head -10` this command is not working – Shabby Jul 22 '22 at 07:20

disk is full, but cannot find big files or folders

Update

9 Answers9

Linked

Related