1

What I want

My actual long term goal is bypassing all the output of a git process into another function (in order to make a progress bar - see my older question for more details about that).

What I've got so far

I thought my problem was solved after this question but it comes out now that only partially.

I have already figured out that git

  • produces output only to stderr by design
  • I had to use the --progress option of git in order to force git to really print out the progress.

Now if I run

                                  #or also &> output.file
git clone --progress http://someRepo > output.file 2>&1

I can see the full progress e.g. by running at the same time in a second terminal

tail -f output.file

I get

Cloning into 'someRepo' ...
remote: Counting objects: 2618, done.
remote: Compressing objects: 100% (14/14), done.
Receiving objects:   4% (106/2618), 42.11 MiB | 5.67 MiB/s

which gets updated in realtime until finishing

Cloning into 'someRepo' ...
remote: Counting objects: 2618, done.
remote: Compressing objects: 100% (14/14), done.
remote: Total 2618 (delta 2), reused 12 (delta 1), pack-reused 2603
Receiving objects: 100% (2618/2618), 258.95 MiB | 5.71 MiB/s, Done.
Resolving differences: 100% (1058/1058), Done.
Checking connectivity ... Done.

So far so well.

My Problem

Now I do not want to put this output into a file anymore but rather bypass it into a function. So I tried

                                  #or also |&
git clone --progress http://someRepo 2>&1 | {
    while read -r line
    do
        # echo is for testing only
        # later I want to pass this to another function
        # EDIT: I added bypassed in order to see what is bypassed
        echo "bypassed: ${line}"
    done
}

But running it like this I only get as output displayed (EDIT: I added the bypassed: to the echo line in order to see how it gets passed.)

bypassed: Cloning into 'someRepo'
bypassed: remote: Counting objects: 2618, done.
remote: Compressing objects: 100% (14/14), done.

and only at the very end when the download finishes I get the rest all at once jumping to this output (EDIT: I added the bypassed: to the echo line in order to see how it gets passed.)

bypassed: Cloning into 'someRepo' ...
bypassed: remote: Counting objects: 2618, done.
remote: Compressing objects: 100% (14/14), done.
remote: Total 2618 (delta 2), reused 12 (delta 1), pack-reused 2603
Receiving objects: 100% (2618/2618), 258.95 MiB | 5.71 MiB/s, Done.
Resolving differences: 100% (1058/1058), Done.
bypassed: Checking connectivity ... Done.

So apparently the 5 lines of the progress block are all passed in at once .. maybe this is a problem for the pipe and the while loop?

So what am I missing? How to I have to run the command in order to get the output also "echoed" / bypassed into my function later in realtime just the same way the tail -f output.file gets updated in realtime?

EDIT

I also already tried stdbuf and even unbuffer as suggested here and here

stdbuf -i0 -o0 -e0 git clone --progress http://someRepo |& ...

stdbuf -oL git clone --progress http://someRepo |& ...

unbuffer git clone --progress http://someRepo |& ...

but this didn't change the output.

Community
  • 1
derHugo
  • 3,306
  • 5
  • 30
  • 49
  • 1
    Perlhaps there's some buffering: try `stdbuf -oL git ...` -- see [`stdbuf(1)`](http://manpages.ubuntu.com/manpages/xenial/en/man1/stdbuf.1.html) man page – glenn jackman Dec 28 '17 at 15:15
  • @glennjackman already tried this (see my EDIT) – derHugo Dec 28 '17 at 15:15
  • Based on this [so] answer (2013 - not sure if things have changed since), it looks like `git` outsmarts you by turning off the error stream if it detects that it is not going to a tty [Trying to redirect 'git gc' output](https://stackoverflow.com/a/18006967/4440445) – steeldriver Dec 28 '17 at 15:55
  • 1
    @steeldriver but isn't that what the `--progress` option of `git` should change? ([man git-clone](https://git-scm.com/docs/git-clone)). It is working perfect when redirecting the output (from `stderr`) into the file. I guess you actually where right about the buffering. But how can I solve this? – derHugo Dec 28 '17 at 16:12
  • You have one problem and add one question after another, each with just snippets of information about your greater goal – why don't you just post *one* question stating the *actual* problem and show what you did *in total* there? – dessert Dec 28 '17 at 16:13
  • 1
    Well as said before in my other questions my main goal is described in [this question](https://askubuntu.com/questions/988403/bash-pass-command-output-to-function-in-realtime). I thought my problem was completely solved in [my other one](https://askubuntu.com/questions/989015/how-to-get-git-producing-output-to-a-file) but actually this only solved getting the stream to a file (which is still usefull) not to a function. This is the last part missing now so I didn't think putting again alltogether in one single question would be helpfull but instead a lot of reading until getting to the point. – derHugo Dec 28 '17 at 16:17
  • 1
    But instead of voting my other questions down now, you could also try to solve this one here... I even provided the complete script I'm working on on gist and posted the link in [my other question](https://askubuntu.com/questions/989015/how-to-get-git-producing-output-to-a-file): https://gist.github.com/Yolo-Jerome/e7e32f5c588f4209b362bff2f120d2e1 <- in this script I want to bypass ouput of all `git` lines into the `printInfo` function in order to get my progressbar working (which it does in other scripts where ALL output is only generated by this `printInfo` function) – derHugo Dec 28 '17 at 16:20

2 Answers2

4

This is a very interesting question and I had to experiment for a while, but the answer is actually super simple!

How does git's progress output work? It shows a single status line with a progress percentage and updates it all the time until it finished. It does that by not printing a line feed \n but a carriage return \r character at the end of the progress line, which causes the output cursor to jump back to the beginning of the line again, ready to overwrite the last written line again with an updated value.

You can observe this by piping git's output through cat -A, which does not interpret invisible characters but shows them, so that e.g. \r becomes ^M and line breaks additionally get denoted with a $:

Cloning into 'MyRepository'...$
remote: Counting objects: 2317, done.        $
Receiving objects:   0% (1/2317)   ^MReceiving objects:   1% (24/2317)   ^MReceiving objects:   2% (47/2317)   ^MReceiving objects:   3% (70/2317)   ^MReceiving objects:   4% (93/2317)   ^MReceiving objects:   5% (116/2317)   
[...]

Now why does this affect read? It's obvious, as help read says (extract, emphasis mine):

Read a line from the standard input and split it into fields.

read waits for line breaks (\n) before it finishes. The output of git did not contain any line breaks during the progress display, hence read did not finish. Only after the progress display was completed and git really printed the next line, read took in all the output, including all intermediate states and carriage returns and you echoed it inside your loop. If you pipe the echo or done through cat -A as well, you see the same output full of ^Ms, like above.

What you can do to allow read to capture all the intermediate progress lines is to translate all carriage returns into real line breaks, by piping the stuff through tr \\r \\n. That way each state gets printed on a new line instead of overwriting the previous line.

So, the complete loop you used could look like this:

git clone --progress http://someRepo 2>&1 | tr \\r \\n |
    while read -r line ; do
        echo "bypassed: ${line}"
    done

On my system, the output shown in terminal by this solution is still not totally fluent and stutters a bit, but I was not able to improve that any further. At least you now have the real "progress" output.

Byte Commander
  • 105,631
  • 46
  • 284
  • 425
  • Sorry until now I'm looking over this. I tried it exactly with your line `2>&1 | tr \\r \\” |` but I don't really understand what this `”` character does. Also now I'm getting not any output from git anymore – derHugo Dec 28 '17 at 18:22
  • 1
    Oooh, oops. That's a typo. `”` should have been a simple `n`. – Byte Commander Dec 28 '17 at 18:39
  • 1
    Ah yes now it makes sence. So now ... using `git clone ... |& tr '\r' '\n'` works -> I get the output line by line instead of inline updated .. **but** as soon as I add the loop `git clone ... |& tr '\r' '\n' | { while ...` it prints out nothing anymore – derHugo Dec 28 '17 at 18:44
  • Ok hm .. now with another repository it worked :D – derHugo Dec 28 '17 at 18:46
  • Did you find out why nothing got printed any more with your old repo @derHugo? I can't imagine any reason other than a typo or maybe missing quotation marks or braces or so... – Byte Commander Dec 28 '17 at 19:01
  • @ByteCommander still no clue and still happens. I only changed the URL and the only difference is: The second repo is a public one (mine) the first is a private repo of my company .. but I don't see how this would matter for the output ... – derHugo Dec 28 '17 at 19:12
  • also tried it now with `git ... |& tr '\r' '\n' | cat -A` just to see .. and it seems the second pipe doesn't receive anything when using the first repo ... on the second one it does as expected – derHugo Dec 28 '17 at 19:29
0

Here is a complete solution, not very nice, but working well !

total=0
{
git clone --progress http:yourRepo 2>&1 | tr \\r \\n |
    while read -r line ; do
        cur=`grep -oP '\d+(?=%)' <<< ${line}`
        total=$((total+cur))
        percent=$(bc <<< "scale=2;100*($total/11767)")
        echo "$percent/1" | bc
    done
} | whiptail --title "Cloning" --gauge "Cloning Git Repository" 8 78 0
clo
  • 1