8

Server is Ubuntu 16.04. I have a process running with nohup that logs into a local file. This file gets big enough to consume 100% disk space. To resolve this I have to kill the process first and then remove the log file. Then I restart the process. How can I resolve this with a script or some other tool?

Zanna
  • 69,223
  • 56
  • 216
  • 327
DEVCNN
  • 183
  • 1
  • 1
  • 6

3 Answers3

14

With logrotate you can configure how big a log file may get or after how much time:

  • the log files are rotated (log.n becoming log.n+1, and the last log file being deleted)

  • the current log file is truncated without disturbing the writing process.

Take a look at man 8 logrotate.

muclux
  • 5,034
  • 4
  • 20
  • 43
  • Maybe this would have worked if the file wasn't open at the time i run logrotate. It tries renaming the file and fails. nohup keeps streaming data into the log file. So i guess it must be open all the time. – DEVCNN Apr 02 '19 at 08:12
  • 3
    @DEVCNN You are learning why using standard output and redirecting it to a log file is just about the **worst** possible approach to logging. Your log file is tied to the logging process, and **you can't do anything about it**. The actual solution is to use a proper logging system that would allow you to separate the actual log file from the process and allow you to use tools like `logrotate`. There are a lot of really good reasons things like `syslog` and `logrotate` exist - and you're learning a lot of them. – Andrew Henle Apr 02 '19 at 11:42
  • Sure. I'll look into that. – DEVCNN Apr 02 '19 at 13:01
4

I guess that you start the script/program with nohup like

nohup scriptname 1>logfile.log 2>& &

I would recommend instead of deleting the log file just to clear it with

echo -n >logfile.log

If you delete/move an open file it will be written until the process will close the file or the process will end.

0x0C4
  • 703
  • 5
  • 12
  • Useless use of `echo`. Just `>logfile.log` – rexkogitans Apr 02 '19 at 10:36
  • using "echo" doesn't hurt. it's build-in – 0x0C4 Apr 02 '19 at 10:38
  • 2
    This is an **extremely** unreliable way to free disk space from a full log file. The process writing to the log file has a current offset, and truncating the file with `>` doesn't change that offset. The next time the file gets written to, you'll likely get either a sparse file if the underlying filesystem supports sparse file, or you'll wind up with a file the same size, almost entirely full of `\0`/NUL characters and taking up the same amount of space. – Andrew Henle Apr 02 '19 at 11:39
  • 1
    @AndrewHenle this answer is simply wrong since the shell is being used for redirection. However, if the redirection is changed to `>>` then it works because the log file is opened with O_APPEND. See https://unix.stackexchange.com/questions/202797/does-bash-open-files-in-o-append-when-using-on-linux. In general though, your warning is correct. All log files should be opened with O_APPEND to make rotation easy but alas some programs don't do that. – Mark Wagner Apr 02 '19 at 18:14
  • @MarkWagner So if i use >> instead of >, can i use logrotate then? Does >> open the log file only when writing? – DEVCNN Apr 03 '19 at 05:02
  • ">>" append data to an already existing file. so if you re-direct output during calling the process again ("nohup script 1>>logfile.log 2>&1") it would append output of stdout and stderr to "logfile.log" and this has nothing to do with the work of logrotate. This is only about how it works with already existing data in a already existing log-file. – 0x0C4 Apr 03 '19 at 06:39
  • Ok. I get it. Seems like I have a lot of reading to do. – DEVCNN Apr 03 '19 at 07:23
  • @DEVCNN If you use `>>` you can use the simple `>logfile.log` method of log rotation without having to restart the process which was the main goal. You can implement rotation with a cron job, logrotate, or probably systemd (what can't it do?). – Mark Wagner Apr 03 '19 at 15:38
0

To delete all logs automatically, edit the .bashrc file using your favorite text editor. Here I'm using nano. In your terminal run:

nano ~/.bashrc -

Add the following to the bottom of the file:

rm -r /var/log   # Deletes logs directory
clear            # Clear the terminal

Press Ctrl+O to save and Ctrl+X to exit edit mode.

The .bashrc file is executed every time you log in or launch a terminal instance, thus your logs will always be deleted.

You can also delete them based on time. E.g. delete all logs created 3 days ago using:

find /yourlog/path -mindepth 1 -mtime +3 -delete
  • -mindepth 1 means process all files except the command line arguments.
  • -mtime +3 will check for the files that were modified 3 days ago.
  • -delete will delete them.
BeastOfCaerbannog
  • 12,964
  • 10
  • 49
  • 77
David Kariuki
  • 198
  • 1
  • 12