20

I'm currently trying to make a systemd unit as a web server. Currently, my foo.service file looks as follows:

[Unit]
Description=The Foo Web Server

[Service]
Type=simple
ExecStart=/opt/foo/.cabal-sandbox/bin/foo

[Install]
WantedBy=multi-user.target

The foo executable automatically logs all HTTP requests to stdout - this is well tested. However, when I view the logs with journalctl -u foo, I only get output like this:

...
May 06 17:46:57 localhost systemd[1]: Stopping The Foo Web Server...
May 06 17:46:57 localhost systemd[1]: Started Foo Web Server.
May 06 17:46:57 localhost systemd[1]: Starting The Foo Web Server...
May 06 17:47:08 localhost systemd[1]: Stopping The Foo Web Server...
May 06 17:47:08 localhost systemd[1]: Started The Foo Web Server.
May 06 17:47:08 localhost systemd[1]: Starting The Foo Web Server...

Could someone explain why it's not logging all stdout output? I looked briefly at this previous question, but it doesn't help - however it alluded to something along the lines of "...may not work for systems that don't use full systemd" - would this be the case for Ubuntu 15.04? Thank you in advance, any help with this would be much appreciated!

Community
  • 1
Athan Clark
  • 607
  • 3
  • 6
  • 9
  • 1
    Make sure, your process is not buffering the output. I was having similar problem and it got resolved by disabling output buffering of my python script. As the number of log records produced was not high, it looked like there is no logging, but in fact, it had yet no chance, as all was still gathering in stdout output buffer. – Jan Vlcinsky May 08 '15 at 20:07
  • 1
    Trying to fix this too. I have an impression that systemd does buffering. stdout is line-buffered in UNIX, but systemd does its own thing and buffers a lot more (in order to be fast but useless maybe). – Velkan May 11 '15 at 11:57
  • 4
    I had the same problem and buffering with Python's print function was the problem! Since I was using Python 3 I just used something like `print('Hello World!', flush=True)` and that did the trick! Output started showing up in journalctl. – timbram Sep 14 '17 at 01:40
  • 3
    For python: an alternative to what @timbram suggested is putting `Environment=PYTHONUNBUFFERED=1` in your service file from https://unix.stackexchange.com/a/608678/161268 – Brad Grissom May 18 '21 at 16:25
  • What strange behavior. Far from ideal. One could even say not very platonic... – Quelklef Sep 13 '21 at 01:38

2 Answers2

12

In fact, buffering in UNIX depends on the context: when stdout is redirected to something interactive like a console - it is usually line-buffered, otherwise it is fully buffered.

Buffering may be changed inside the application using setvbuf library call.

But it can also be done with stdbuf command on launch:

ExecStart=/usr/bin/stdbuf -oL /opt/foo/.cabal-sandbox/bin/foo

(for line-buffered case)

Velkan
  • 3,516
  • 4
  • 24
  • 45
  • This saved my day! Thank you very much. But I'm a bit confused that with `ExecStart=/my/foo/program`, why the stdout log is not flushed when the service is terminated, but completely disappears instead. – wlnirvana Aug 15 '18 at 05:24
  • Unfortunately, the log messages are tagged with `stdbuf` instead of the actual program, but I can confirm that the command-line workaround does work. I'll probably need the library function to fix the tag. – campkeith Aug 08 '22 at 01:22
0

By default on Ubuntu 15.04, systemd journals are only volatile and kept in /run/systemd/journal and are lost at each reboot. To use persistent systemd journal, you need to create the /var/log/journal directory (and restart systemd-journald.service).

So, may be, stdout output is just redirected to syslog and not kept in systemd journal. For that, you may need to use a persistent systemd journal as explained above.

Have you checked /var/log/syslog for your foo log ?

solsTiCe
  • 9,071
  • 3
  • 36
  • 64
  • I created the `/var/log/journal` as you mentioned, but I'm not seeing stdout of my systemd service anyway. in the `/var/log/syslog` I see neither it. – logoff Jul 28 '15 at 07:46