3

How do I execute a command for each line of stdin immediately?

Example:

(echo abc; sleep 10; echo def;) | ???

The xargs tool does not do the job because it blocks until it sees all the data (waits for stdout to close). Yet I have to execute commands immediately as new lines become available.

VasyaNovikov
  • 3,366
  • 3
  • 25
  • 33
  • 1
    http://unix.stackexchange.com/questions/25372/turn-off-buffering-in-pipe might be interesting to read. – Hennes Dec 22 '15 at 10:19
  • @Hennes Thanks. Unfortunately though, it does not help. The problem is not in the pipe, but in the way how `xargs` works (it waits/blocks until it sees all the input). – VasyaNovikov Dec 22 '15 at 10:26
  • 1
    So, do not use xargs. Pass unbuffered output though a pipe and on the other end of the pipe use a shell script to read lines until EOF. – Hennes Dec 22 '15 at 10:27
  • 1
    @Hennes I wrote an example answer to understand the idea behind your comments. – VasyaNovikov Dec 22 '15 at 11:06
  • "The `xargs` tool does not do the job because it blocks […]" – It depends on options in use (e.g. try `xargs -L 1`). – Kamil Maciorowski Feb 13 '23 at 20:13
  • @KamilMaciorowski According to `man xargs`, the `-L` flag sets the maximum lines to be processed. This is very different from processing an infinite amount of lines, but immediately as they come available. – VasyaNovikov Feb 14 '23 at 08:13
  • 1
    At most *n* nonblank input lines *per command line*, not in total. `(echo abc; sleep 10; echo def;) | xargs -L 1 echo input:` – Kamil Maciorowski Feb 14 '23 at 08:23
  • @KamilMaciorowski Thanks for the comment, I can confirm it works! – VasyaNovikov Feb 14 '23 at 10:13

2 Answers2

6

Use a shell script:

long_running_command | (
  while read -r LINE; do
    echo "line is: $LINE";
    command_to_execute "$LINE";
  done;
)

In contrast to parallel, this doesn't have an overhead of ~ 150 milliseconds + 3..7 milliseconds per line. It also works for executing multi-line bash commands, re-using functions or variables.

The idea is based on the comment by @Hennes, finished and written down as an answer. Kudos to him/her!

VasyaNovikov
  • 3,366
  • 3
  • 25
  • 33
4

Using GNU Parallel:

(echo abc; sleep 10; echo def;sleep 10; echo def;)|parallel -uj1

-u is needed to get the output immediately. Without, the command will run, but output will only be printed later.

-j1 forces running a single jobslot. Leaving it out will make GNU Parallel read n jobs before starting, where n = number of cores.

slhck
  • 223,558
  • 70
  • 607
  • 592
Ole Tange
  • 4,529
  • 2
  • 34
  • 51