7

This first pipeline works fine (printing "c"):

echo "a" | sed 's/a/b/' | sed 's/b/c/'

This one does not do what I expect (nothing gets printed when I feed an "a" into my fifo ):

mkfifo fifo;
cat fifo | sed 's/a/b/' | sed 's/b/c/'

However, if I remove the second "sed" command from the latter pipeline, I do get a "b" printed. I think my understanding of pipes and redirects must be too simplistic. Can someone explain to me how to fix the 2nd case so that I can run two successive commands on the contents of the fifo?

(note this isn't a problem specific to fifo, the same behavior occurs with netcat too. I posted a similar question about netcat but got no answers)

Community
  • 1
Sridhar Sarnobat
  • 1,395
  • 2
  • 13
  • 25

1 Answers1

8

Use the -u (--unbuffered) option of sed, so it writes it's output, instead of collecting it first for a fast, large write later.

cat fifo | sed -u 's/a/b/' | sed -u 's/b/c/'

cat is always unbuffered.

If it's not about sed, but some program with output buffering, but without options to disable it, there are tools that can help to force buffering off - see man unbuffer and man stdbuf.

If it's a shell script, it should be unbuffered by default: when a command creates output in the script, and exits, the output is flushed. Otherwise, you use unbuffer inside the shell script.

For background, see this overview of stdio buffering.

Volker Siegel
  • 1,504
  • 11
  • 21
  • Suppose instead of sed, I had my own custom script. What would I do in this case to avoid the buffering? Or should it be unbuffered by default? – Sridhar Sarnobat Aug 04 '14 at 06:42
  • Just some more followup - if I wanted to use perl regular expressions with buffering, it would go something like `nc -k -l 4458 | perl -ne 's{HTTP}{FTP}g; $|++;print $_' | perl -ne 's{FTP}{RTSP}g; $|++;print $_' ` – Sridhar Sarnobat Aug 05 '14 at 04:14
  • Yes, what do you mean? Is there a problem, or is it an example? Oh, and you can use `ssed -R` if you like `sed` with perl regular expressions. – Volker Siegel Aug 05 '14 at 04:32
  • Sorry, that was not a question. I apologize for not being clear, it was just another example of the same concept. Unfortunately it's too late for me to edit that comment. I didn't know sed could understand Perl regex. Perhaps I should start using that instead from now on. – Sridhar Sarnobat Aug 05 '14 at 04:35
  • Oh, no problem! – Volker Siegel Aug 05 '14 at 04:36