49

I am redirecting STDOUT and STDERR to a single file, always growing, so I use always use "append" redirection; this is >>

my command is command >> logfile 2>&1

And it works.

But the STDERR redirection has a single >, which I was using to "create" the file, erasing the previous one, as in command > outlog 2> errlog

Why it does not erase the log file in this case?

Peter Cordes
  • 5,681
  • 1
  • 28
  • 33
Mikel Vergy
  • 591
  • 4
  • 6
  • 9
    Welcome to Super User and congrats on an interesting first question! – gronostaj May 06 '19 at 11:04
  • 14
    The tokens aren't split up the way you think they are. That is, it's not "`>` for redirect to file" and "`&1` for which file to redirect to", it's "`>&` for redirect to file descriptor (which is not the same thing as a file)" and "`1` for which descriptor to redirect to". So it's a completely different operation, and you shouldn't a priori expect to be able to transfer any intuition from `>` or `>>` to `>&`. (This is just a first approximation. In the second approximation, there are some implementation details shared between the three operators, so you *can* transfer some knowledge...) – Daniel Wagner May 06 '19 at 17:49
  • 8
    @DanielWagner Please don't answer in comments. – David Richerby May 07 '19 at 15:38

2 Answers2

60

When you redirect something to &number, you are not opening a new file at all; you're reusing an already open file along with whatever mode it was opened.

The numbers refer to "open file" handles (file descriptors). So there is no technical difference between how >& and >>& (and indeed <&) would work – they all just mean "clone the existing file descriptor using dup()".

That is, 2>&1 indicates that file descriptor #1 (which you previously opened for appending using >>logfile) is cloned into number #2. And yes, 2<&1 works identically.

Side technical note: Appending vs truncating is not an explicit action done by the shell; it's actually a mode the shell specifies when opening the file, and the rest is performed by the OS itself. For example, when you use > the shell doesn't manually erase the old contents, it just adds O_TRUNC when calling open(). Therefore, when open() isn't called at all, the previous mode remains unchanged.

u1686_grawity
  • 426,297
  • 64
  • 894
  • 966
  • Is O_TRUNC something you run into when doing windows programming in C? – barlop May 06 '19 at 11:28
  • 7
    @barlop: Yes; not _exactly_, but quite similar. If you develop on raw Win32 API, the CreateFile() function has TRUNCATE_EXISTING, which means the same thing as POSIX open()'s O_TRUNC, and you have DuplicateHandle() which is like dup(), even if Windows' stdin/stdout are a bit weirder and more special-cased. (If you develop on Mingw, you just get the same POSIX open(). And most higher level I/O libraries, like C stdio or Python io, have modes such as "w" "a" "r+" which directly translate to the same thing.) – u1686_grawity May 06 '19 at 11:38
  • @grawity, your explanation suggests that `command 2>&1 >> logfile` will overwrite the file, which is not the case, could you elaborate that? In my experiment error output went to console and standard output went to the file. – Oliv May 07 '19 at 09:53
  • @Oliv: Does it? You're not using any "overwrite" redirection operator, you're just using the same "duplicate fd" and "append to file". – u1686_grawity May 07 '19 at 10:40
  • 1
    Note that "cloning" here is not "aliasing". Once 2>&1 clones fd 1 to fd 2, they're still two independent file descriptors that merely reference the same "open file". Further redirections of fd 1 do not affect fd 2. – u1686_grawity May 07 '19 at 10:50
  • I think what it does is that it first redirects error output to console and then standard output to console. So the order matters. – Oliv May 07 '19 at 14:43
  • 1
    Fun fact: `O_APPEND` is a special flag that's separate from just omitting `O_TRUNC`. It means any write automatically happens at the end of the file, even if *other* writers made the file longer since your last write. i.e. every `write()` becomes like an atomic combination of `lseek(SEEK_END) + write()`. You can open a file for overwrite without truncation using for example `dd of=foo conv=notrunc` – Peter Cordes May 07 '19 at 21:02
  • 1
    Also, note that order is important... `2>&1 >> logfile` will redirect `stderr` to the terminal, and then subsequently redirect the application's `stdout` to `logfile`. – Attie May 07 '19 at 21:11
14

The sequence command >> logfile 2>&1 has two redirection stages:

  • command >> logfile will append to the logfile
  • 2>&1 will redirect stderr to stdout (which itself appends to the logfile)

So command >> logfile 2>&1 will not truncate the log file, while command >>logfile 2>logfile would.

Eugen Rieck
  • 19,950
  • 5
  • 51
  • 46
  • 1
    What about `command >>logfile 2>>logfile`? – Zereges May 06 '19 at 12:36
  • @Zereges Probably gives an error because you try to open logfile for writing twice. One of the 2 "opens" should fail with an error along the lines of "file already open (or locked) by another process" – Tonny May 06 '19 at 22:39
  • 7
    No, files on Unix-like systems are not locked merely by being open. (For that matter, e.g. Linux doesn't support mandatory locking _at all._) – u1686_grawity May 07 '19 at 07:14
  • 1
    @Zereges: You'd get 2 separate file descriptors (not duplicates of each other) each with their own file position. But they're both opened in `O_APPEND` mode, so every write is like an atomic `lseek(SEEK_END) + write()`. So I think it would Just Work with the same end result as normal `dup2()` but probably slightly less efficient. – Peter Cordes May 07 '19 at 21:05