tail a log file but show only specific lines

Question

I'm tailing a log file with -f flag. Then I'm piping this to grep, to find only lines that contain "X". That's working perfectly fine. Now I want to pipe this again into another grep, that will remove all the lines containing "Y". When I add the second pipe, the file stop refreshing and it looks like no data is coming.

This is the command that works: tail -f my_file.log | grep "X"

This is the command that doesn't: tail -f my_file.log | grep "X" | grep -v "Y"

How should I structure this so that the command works?

try to do one pipe by one pipe, change the sequence, do `tail -f file|grep -v "Y"` . if the output is ok then proceed to append `grep "X"`. — Aizuddin Zali, Oct 13 '15 at 13:16
[How to 'grep' a continuous stream?](https://stackoverflow.com/q/7161821/995714) — phuclv, Jan 18 '18 at 08:07

score 62 · Accepted Answer · answered Oct 13 '15 at 13:28

62

As the output of grep is buffered, use --line-buffered option of grep to enable line buffering:

tail -f /path/to/log | grep --line-buffered 'X' | grep -v 'Y'

If your grep does not have the option, you can use stdbuf as an alternative:

tail -f /path/to/log | stdbuf -oL grep 'X' | grep -v 'Y'

answered Oct 13 '15 at 13:28

heemayl

90,425
20
200
267

1

I am wondering how `stdbuf` tells `libstdbuf.so` which settings to use. – kasperd Oct 14 '15 at 07:21
@kasperd: Environment variables. – Nate Eldredge Oct 14 '15 at 18:47
1

@NateEldredge I already looked for environment variables in the output from `diff -u <(env) <(stdbuf env)` and found none. But now I realize that what I should have tested was `diff -u <(env) <(stdbuf -oL env)`. – kasperd Oct 14 '15 at 19:59
I am also same sort of problem. my case is, I need to print all lines contain 'aaa' and 'bbb'. Above first solution not working. Second solution is working for 'aaa' exist and 'bbb' not exist. both exist not working. giving no output. my command looks like this: `tail -f test.txt | stdbuf -oL grep 'aaa' | grep 'bbb'` it not giving any output. Could you please help me. – Sun Apr 18 '17 at 04:54

fedorqui · Answer 2 · 2015-10-13T14:50:31.423

22

I normally find more useful awk for these kind of logical checks:

tail -f /path/to/log | awk '/X/ && !/Y/'
#                           ^^^    ^^^^
#                   this I want    but not this

Tested by having two tabs, one in which I keep writing seq 20 >> myfile and the other one having for example tail -f myfile | awk '/3/ && !/13/'.

edited Oct 13 '15 at 14:50

answered Oct 13 '15 at 14:42

fedorqui

9,909
1
23
41

score 15 · Answer 3 · answered Oct 13 '15 at 14:36

Another approach would be to use a single grep invocation instead of two and so avoid the buffering issue. Just use a regular expressions that matches lines consisting of 0 or more non-Y characters, then an X and then 0 or more non-Ys again till the end of the line"

tail -f /path/to/log | grep '^[^Y]*X[^Y]*$'

tail a log file but show only specific lines

3 Answers3