Re: How to use Bash pipefail with GNU Parallel

[Ole Tange]

On Mon, Nov 23, 2015 at 11:25 PM, David Ventimiglia <dventimi@gmail.com> wrote:

> How do I use Bash pipefail with GNU Parallel?
:
> I want any failed pipeline command to cause GNU parallel to fail (i.e.,
> return non-zero exit status).
>
> Here's an example of a command stored in the database table.
>
>> curl -sf <some URL> | awk <some program>:
:
> The problem is that
> the individual commands never fail (i.e., they don't return a non-zero exit
> status) because the last command in their pipelines--the call to awk--never
> fails.
:
> Evidently, Bash has a convenient facility for this.  One can set the
> 'pipefail' option so that the exit status of pipelines is right-most
> non-zero exit status of its constituent commands, or zero if none of its
> commands fails.  That's exactly what I want.  The question is this.
>
> How do I pass 'set -o pipefail' to the commands that are run by GNU
> parallel?
>
> I think the answer lies somewhere in the labyrinth of combinations of using
> some GNU Parallel options (-q --shell-quote -I{}) and some Bash options
> (-c).
>
> Here's what I tried so far, and which didn't work.
>
>> sql -n <some database URL> 'select command from mycommands' | parallel
>> --halt 2 bash -c 'set -o pipefail; {}'

You are being hit by quoting and your thinking of -q is correct. -I{}
will not help you, and --shell-quote is used for quite a different
problem.

    sql -n <some database URL> 'select command from mycommands' |
parallel -q --halt 2 bash -c 'set -o pipefail; {}'

Alternatives:

    myfunc() {
      bash -c "set -o pipefail; $1"
    }
    export -f myfunc
    ... | parallel --halt 2 myfunc

or:

    ... | parallel -q --halt 2 bash -c "'set -o pipefail; {}'"

or make a script:

    #!/bin/bash
    set -o pipefail
    "$@"

and call that:  ... | parallel myscript {}

Personally I have grown to like functions: They are easy to test
before giving them to GNU Parallel, they are easy to read, they can be
put in the same file/script, they can be exported to remote hosts
(using --env), and (most importantly) they do not require a forest of
mixed '"\ which can be a hell to debug.


/Ole

PS: Just for fun:

fun() {
  echo "$1" is fun;
}
export -f fun
parallel -S localhost --env fun -vv fun ::: this