1

I have a command I am running with curl. The curl command always resolves successfully even when the server response includes the string ERROR somewhere.

Can I somehow detect this response with some sort of shell script or function (zsh preferred), while also piping the output through to the terminal, and making the overall command falsy?

Alan H.
  • 2,758
  • 8
  • 27
  • 39
  • 2
    Is it the *string* that really matters? Please see [this answer](https://superuser.com/a/1690892/432690) and links therein. – Kamil Maciorowski Apr 23 '22 at 02:17
  • In my case, yes. The `curl` command is used to trigger a remote script that may encounter an error, but the HTTP response is considered successful (200) regardless – Alan H. May 05 '22 at 15:49

2 Answers2

2

I don't knows exactly what you want to achieve, but I guess you need either tee or a redirection of the error output curl -s whatever 2>&1|tee output.txt|grep ERROR

so you both have the grep of the error and the contain of the curl command in the file output.txt.

dominix
  • 176
  • 7
  • i don’t want to save output to a file, just have it show up in the terminal as it does with a naked `curl` call – Alan H. Apr 23 '22 at 06:21
2

Please see the first part of this other answer of mine and links therein. You want to detect a string. Making curl fail according to HTML error code seems more elegant. If it's possible to get the desired result this way then it's the Right Way.

Otherwise use the following general-purpose shell function:

failon() (
  pattern="$1"
  shift
  [ "$#" -eq 0 ] && set cat
  FAILON_STATUS="${FAILON_STATUS:-125}"
  set -o pipefail
  { "$@" \
    | tee -p /proc/self/fd/3 \
    | { grep -q -- "$pattern" && return "$FAILON_STATUS" || return 0; }
  } 3>&1
)

Usage:

failon [pattern [command [arg...]]]

The main purpose of the function is to return non-zero exit status if the output of command [arg...] contains a line matching the pattern. The output is printed to the stdout of the function, regardless of whether there is a match or not.

Example:

failon ERROR curl …

Notes and explanations:

  • The pattern is supplied as-is to grep, so it's a regex. An empty or undefined pattern will match any line, if only there is a line.

  • The function should work in bash, zsh and many other shells. AFAIK the shell syntax I used is portable. Non-portable things are independent from the shell, except set -o pipefail which belongs to the shell and it's not portable. I mean not yet. It's widely supported and it's going to be added to the POSIX standard.

  • Thanks to pipefail, if the pattern is not found and tee doesn't fail (normally it shouldn't fail) then the function will return the exit status of the specified command (e.g. curl).

  • If the pattern is found then the function will return 125. I chose 125 because:

    • it's not reserved,

    • man 1 curl in my OS specifies exit codes up to 96 already and more may appear in the future.

    You can adjust this number to your needs via FAILON_STATUS environment variable. E.g. FAILON_STATUS=120 failon …. Note FAILON_STATUS=0 is technically valid but rather useless.

  • tee -p is not portable. Without -p tee will exit if it gets SIGPIPE after grep exits early. grep -q exits as soon as it finds the pattern. Thanks to -p tee will relay its whole input to /proc/self/fd/3 (which is ultimately the stdout of the function), even if grep exits early.

    If you tee does not support -p, the most straightforward workaround is to use grep -- "$pattern" >/dev/null. Now grep will silently process all the data, it will not exit early. The downside is it will do unnecessary job matching lines after the pattern is first found.

  • /proc/self/fd/… is not portable; hopefully your OS supports it. Even if it doesn't, there's always a method of curl … | tee /temporary/file followed by ! grep -q ERROR /temporary/file to get falsy exit status upon ERROR. This method is quite straightforward and KISS, but:

    • in one of your comments you said you don't want to save the output to a file;

    • in general it's not easy to safely and reliably create a /temporary/file; mktemp is the right tool, but it's not specified by POSIX.

  • In zsh you can do "$@" >&3 | { grep -q … (instead of "$@" | tee … | { grep -q …) and thus get rid of tee -p /proc/self/fd/3 and any potential problem from it.

  • -- is explained here: What does -- (double-dash) mean?

  • If command is not specified then the function will use cat. This allows you to use failon as a "filter":

    curl … | failon ERROR

    Note it's easy to lose the exit status of curl this way. On the other hand it's possible to do this:

    curl … | failon ERROR | failon WARNING

    and examine PIPESTATUS (in bash) or pipestatus (in zsh) to detect ERROR and WARNING in the output independently.

  • Some programs change their behavior, depending on stdout being a terminal, a pipe or a regular file (see how to trick a command into thinking its output is going to a terminal). Our function internally uses a pipe, so in some cases the output from failon foo program will be different than the output from the same program invoked directly.

Kamil Maciorowski
  • 69,815
  • 22
  • 136
  • 202
  • Just an incredible answer. Hats off. ··· Regarding a temp file, that is fine if it helps make the code a little more robust/KISS/portable. I was mostly just thinking that I didn't need to save the response for any (other) purpose. Me personally, I’m on macOS + zsh. As you mention, `tee -p` isn't portable and doesn’t work for me, but I was able to use the substitution you recommended. Thanks so much – Alan H. May 05 '22 at 15:48
  • Just to check, is it at all possible that using this like `failon error curl ...` might cause the `curl` command to fire off twice in parallel? – Alan H. May 05 '22 at 17:49
  • @AlanH. Not by design; and I cannot spot a bug either (for now). But I know `curl` can do some things in parallel (depending on some options); maybe this involves spawning a child `curl`, I really don't know. If you tell me there are two `curl` processes whose parent is the shell then it will get interesting. – Kamil Maciorowski May 05 '22 at 17:59
  • Right. Well, i think somehow using `failon` is causing my `curl` command to be changed. I’m using a form like `curl -X POST -d 'foo=bar/baz' https://url.example.tld/script` — can you imagine how that might possibly be mangled? – Alan H. May 05 '22 at 18:39
  • @AlanH. Few ideas: (1) Is your `curl` an alias? (2) Is Windows involved in running your `curl`? (e.g. your work in Cygwin or so). // `"$@"` (after `shift`) should run the `curl …` part of `failon error curl …` as if you typed it as `curl …`. The difference is `curl` prints to a pipe. You wrote "`tee -p` isn't portable and doesn’t work for me, but I was able to use the substitution you recommended". What exactly did you do and in what shell? – Kamil Maciorowski May 05 '22 at 18:57
  • `which curl` → `/usr/bin/curl` (no alias afaict). No Windows involved. I modified the line that included `tee -p` to read instead: `| "$@" >&3` (with trailing backslash before the newline) – Alan H. May 05 '22 at 19:04
  • It may all become a moot point because the dev who maintains the script on the server is going to change it to return an error status when the script fails instead of always returning 200… I feel bad now because you’ve put so much impressive effort in here, but maybe it will help some others! And I learned some things from your answer! – Alan H. May 05 '22 at 19:06
  • 1
    @AlanH. Wait. Are you doing `"$@" | "$@" >&3 | …`? Then you're invoking `"$@"` twice; i.e. yes, you're invoking `curl` twice. Let me rebuild the relevant part of the answer. – Kamil Maciorowski May 05 '22 at 19:09