25

The disk that contains Ubuntu on my computer is 115GB in size.

When the disk was 114.7GBfull.

I started deleting some files here and there to free up around 5GB.

But I noticed the disk was 114GB filled again. I thought it was some cache or swap thingy so didn't give it much thought and went ahead and freed around 40GBby shifting some media files out of the disk.

I have a notification in an hour or so that the disk is full yet again!! 40GB!!! All gone!

So I debugged the problem down to the /var/log/syslog file which was initially definitely less than 20GB. I know this because /var folder was 17GB before I cleared the memory.

/var/log/syslog file is currently 55.9GB!

Could someone be so gracious to explain this anomaly? Is this a bug? Or am I affected with some malware or virus?

ant_1618
  • 731
  • 2
  • 7
  • 11
  • are you using a VPN by chance? – Kalamalka Kid Mar 16 '16 at 00:39
  • 1
    It depends what all apps and stuff running in your system? You can clean some of the old/archived entries under /var/log(recursively) which are *.gz. But you watch under /var/log in terms of why some of the logs are growing so fast? – Ashu Mar 16 '16 at 00:54
  • Ashu is right... Perhaps edit your question to tell us what entries in the log file are causing it to grow so quickly – Kalamalka Kid Mar 16 '16 at 01:19
  • 1
    Well, have a *look* in /var/syslog, and see what is filling it up! – psusi Mar 16 '16 at 02:51
  • 1
    +1 ... you might want to use something like `tail -f /var/log/syslog` from a terminal (to avoid having to load such a large file into a text editor) – steeldriver Mar 16 '16 at 07:04
  • @KalamalkaKid Yes I am but I have been using it for a long time now. – ant_1618 Mar 17 '16 at 09:43
  • 1
    Yeah I checked the file it is the error of the wifi monitor `mon0` I ran. Cant believe it flooded 50GB. The same error XD – ant_1618 Mar 17 '16 at 09:45
  • I asked that because brute force attacks can often lead to huge log files, but those are usually different logs. It looks like you found the culprit though. You may want to ask a new question about the wifi monitor mon0 problem to see how to get that solved. – Kalamalka Kid Mar 17 '16 at 09:54
  • I see @KalamalkaKid. Yeah you are right, have to see what mon0 leads to. Thanks :) – ant_1618 Mar 17 '16 at 10:03
  • 2
    Possible duplicate of [Very large log files, what should I do?](https://askubuntu.com/questions/515146/very-large-log-files-what-should-i-do) – karel Feb 24 '19 at 00:12
  • One possible cause for syslog quickly filling up disk space: nvidia driver + VLC. See: https://unix.stackexchange.com/questions/561565/ – Attila Csipak Jun 28 '20 at 14:10

6 Answers6

30

This indefinite growth generally occurs due to repeated log of one or more errors from the same source. In my case it was due to continual report of connection error from the wifi monitoring interface mon0 I have used to monitor my wifi traffic. There have been error reports of such overflow occurring in various other interfaces like tun0 from VPN etc.

I have resolved my issue by clearing the /var/log/syslog file

To tackle this error

  1. You need to find the source of this error and stop it from producing any further overflow of log
  2. Then clear the var/log/syslog file

Problems you might face doing the same

  1. Cant open var/log/syslog: due to massive size any editor is bound to crash
  2. Cant clear /var/log/syslog: Again due to massive size clearing is a challenge too

So, for viewing the error that caused the overflow

tail -f /var/log/syslog

For clearing use:

sudo cat /dev/null > /var/log/syslog
ant_1618
  • 731
  • 2
  • 7
  • 11
11

I had similar issues, my syslog file had 115GB and syslog.1 another 115G, plus multiple compressed files.

1st step find the source:

watch tail /var/log/syslog

you will probably notice the common erro entries; after that; assuming that your files are too big; it is ALMOST pointless to rotate. So, you can delete all the compressed files and files *.1 to recover disk space (my case about 300GB)

2nd step TRUNCATE THE FILE, DO NOT DELETE (or you may have a lot of problems with permissions in the future), there are many methods, including:

sudo tee /var/log/syslog </dev/null

you can even do the second step before and keep watching to find the cause, but be sure, if you don't it will happen again. Probably it is something in a loop, system services is a good place to start looking (something that restart very fast and always for exemple)

Thiago Conrado
  • 368
  • 3
  • 5
  • I don't know why but `sudo tee /var/log/syslog /var/log/syslog` work too but didn't work :/ – uzay95 Jul 29 '20 at 08:03
3

Check the /var/lib/logrotate/status and make sure that it's getting rotated properly. You also need to view the contents of the file and see if it's a system issue throwing alarms constantly.

Tom Damon
  • 477
  • 3
  • 8
3

Try this. It should work correctly and clean it up:

sudo sh -c 'cat /dev/null > /var/log/syslog'
anonymous2
  • 4,268
  • 7
  • 33
  • 61
Ahmed Kamel
  • 39
  • 1
0

Step 1: Create a script file example clearlog.sh in location /var/log/ with file permission 'chmod 775 clearlog.sh'

clearlog.sh file with below lines:

#!/bin/sh
LINECOUNTS=(`cat /var/log/syslog | wc -l`)

if [ $LINECOUNTS -gt "1000" ];
    then
        echo "Recreating syslog file due to lines "$LINECOUNTS" are more than 1000 lines"
        truncate -s 0 /var/log/syslog
        sleep 5
else
        echo "syslog file having line count "$LINECOUNTS" less than 1000}"
fi
/bin/systemctl restart syslog
exit

Step 2: Edit the file in location /etc/logrotate.d/logs add postrotate section

/var/log/syslog {
        daily
        rotate 3
        missingok
        copy
        notifempty
        compress
        dateext
        dateformat -%Y%m%d%H%M
        postrotate
                /var/log/clearlog.sh
        endscript
}

To test above script from command terminal execute this cli 'logrotate -f /etc/logrotate/logs'

Manikandan Jayaraman
  • 1
  • 1
0

A late response to the original post but this helped me. When running a program as a service it seems that all printfs are diverted to syslog rather than stdout, if you have a lot of printfs then this will fill up the syslog very quickly. I had some issues with some of the methods suggested here but

sudo tee /var/log/syslog </dev/null

worked every time.

Note that (for me) I when the syslog file filled up enough it prevented the pi desktop from starting up - I worked around this by logging in through Putty and using the tee command first then rebooting.

Steve M
  • 1
  • This is absolutely the wrong way to handle the problem. The logfile is filling up fora reason - find out why and fix the problem(s). Deleting (old) logfiles can be a work-around ubtil the real problem is fixed. Deleting / emptying the active logfile (here /var/log/syslog) will normally not release the disk-space before the file is closed by restarting the syslog service or rebooting the machine. – Soren A Aug 10 '22 at 08:29