Re: Pipe status codes

[David Erickson]

>On Thu, Mar 29, 2012 at 6:02 PM, Ole Tange <tange@gnu.org> wrote:
> On Thu, Mar 29, 2012 at 9:23 PM, David Erickson <halcyon1981@gmail.com> wrote:
>> Hi there-
>> I just discovered parallel and it looks great.
>
> Good to hear. How did you discover it? What could I have done to make
> you discover it earlier?

I found it through one of the comments in this thread:
http://stackoverflow.com/questions/463963/parallel-processing-from-a-command-queue-on-linux-bash-python-ruby-whateve

I'm not sure what the key words you use on your website are, I believe
I tried searching for bash job queue to begin with and it led me to
the above.  I'm surprised it doesn't come up in any conversation
relating to a tool like this because this project has about a zillion
more features than any of the others I ran in to in the same category.

>
>> One existing use case
>> I have for commands I would like to run is something like:
>>
>> parallel 'cmd1 {} 2>&1 | tee somewhere.log'
>>
>> I have a couple questions regarding this use case:
>>
>> 1) Will parallel correctly honor my request to redirect stderr to stdout?
>
> This is easy for you to test, so I will leave that for you as an exercise.
>
>> 2) Is there some way to use --halt with the error code returned from
>> cmd1?  IE in bash I can do this with ${PIPESTATUS[0]}, because tee
>> will always return 0.
>
> GNU Parallel looks at the exit code, so you can do:
>
> parallel 'cmd1 {} 2>&1 | tee somewhere.log; exit ${PIPESTATUS[0]}'
>
> You can combine this with --halt if you need GNU Parallel not to spawn
> more jobs. If you just need to know how many jobs failed, the default
> (--halt 0) will work for you.

Perfect, with --halt 2 is what I'm looking for.

>
>> Alternatively, can parallel log both stdout and
>> stderr for each command to a unique file with naming of my choice?
>
> You probably want one of these:
>
>  ... | parallel cmd1 {} > all_stdout 2> all_stderr
>  ... | parallel --tag cmd1 {} > all_stdout 2> all_stderr
>  ... | parallel cmd1 {} \> {}_stdout 2\> {}_stderr
>  ... | parallel cmd1 {} \> {#}_stdout 2\> {#}_stderr
>
> But more creative ways exists. If every 3rd argument is the
> stdout-name and the next is the stderr-name:
>
>  ... | parallel -N3 cmd1 {1} \> {2} 2\> {3}
>
> If you have a file with the names of the stdout-names and another file
> with the stderr-filenames:
>
>  ... | parallel --xapply cmd1 {1} \> {2} 2\> {3} :::: -
> stdoutfilenames stderrfilenames
>
> Also you may find --joblog useful.

I'm a little confused by these, what I'd like is the stdout and stderr
to be combined into a single file for each job, where I can supply the
name of that file per job (something like {}.log or similar).  Doable?

Thanks!
David