[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: retry-failed and results directory
From: |
Ole Tange |
Subject: |
Re: retry-failed and results directory |
Date: |
Tue, 16 Oct 2018 20:14:30 +0200 |
On Mon, Oct 15, 2018 at 9:47 PM Daniel LaFlamme
<misterlaflamme@yahoo.com> wrote:
> Support I run a job like this:
>
> parallel --slf hosts.txt --results output-xyz --joblog output-xyz/jobs.log
> --jobs 5 ...
>
> [...] 4 have non-zero status code indicating that they failed. I wanted to
> re-run those and get the results for this second run put in
> output-xyz/1/PARAM1/2/PARAM2/... just like during the first run. However,
> when I re-run like this:
>
> parallel --slf hosts.txt --results output-xyz --joblog output-xyz/jobs.log
> --jobs 5 --retry-failed
>
> I see that the failed jobs run, but their output is not put in nicely
> formatted output-xyz/1/PARAM1/2/PARAM2/...
I can reproduce your problem:
$ parallel --retry-failed --joblog log --results res echo
{}\;exit {2} ::: a ::: 0 1
a 0
a 1
$ find res
res
res/1
res/1/a
res/1/a/2
res/1/a/2/1
res/1/a/2/1/stderr
res/1/a/2/1/seq
res/1/a/2/1/stdout
res/1/a/2/0
res/1/a/2/0/stderr
res/1/a/2/0/seq
res/1/a/2/0/stdout
$ parallel --retry-failed --joblog log --results res echo
{}\;exit {2} ::: a ::: 0 1
a 1
$ find res
res
res/1
res/1/echo a 1;exit 1
res/1/echo a 1;exit 1/stderr
res/1/echo a 1;exit 1/seq
res/1/echo a 1;exit 1/stdout
res/1/a
res/1/a/2
res/1/a/2/1
res/1/a/2/1/stderr
res/1/a/2/1/seq
res/1/a/2/1/stdout
res/1/a/2/0
res/1/a/2/0/stderr
res/1/a/2/0/seq
res/1/a/2/0/stdout
This is due to the design: --retry-failed reads from the log, but the
arguments are not marked in the log, so it is impossible to tell what
was an argument and what was the command template.
While I see your problem, I do not see a way to redesign the log to
support this, especially because you can have named replacement
strings ('--header :') and these names will be part of the dir
structure:
$ parallel --header : --results res echo {arg1} ::: arg1 a b
a
b
$ find res
res
res/arg1
res/arg1/b
res/arg1/b/stderr
res/arg1/b/seq
res/arg1/b/stdout
res/arg1/a
res/arg1/a/stderr
res/arg1/a/seq
res/arg1/a/stdout
So I think we will just have to conclude that --results is
incompatible with --retry-failed. Maybe we should simply issue a
warning, if you use both? What do you suggest?
What you can do instead is to use '--resume-failed'.
--resume-failed cares about the Exitval from the joblog, but only
looks at Seq to figure out which commands to run. This means you can
change the command, but not the arguments. It will run the failed seqs
and the seqs not yet run:
$ parallel --resume-failed --joblog log --results res echo
{}\;exit {2} ::: a ::: 0 1
a 0
a 1
$ find res
res
res/1
res/1/a
res/1/a/2
res/1/a/2/0
res/1/a/2/0/stderr
res/1/a/2/0/seq
res/1/a/2/0/stdout
res/1/a/2/1
res/1/a/2/1/stderr
res/1/a/2/1/seq
res/1/a/2/1/stdout
$ parallel --resume-failed --joblog log --results res echo
{}\;exit {2} ::: a ::: 0 1
a 1
$ find res
res
res/1
res/1/a
res/1/a/2
res/1/a/2/0
res/1/a/2/0/stderr
res/1/a/2/0/seq
res/1/a/2/0/stdout
res/1/a/2/1
res/1/a/2/1/stderr
res/1/a/2/1/seq
res/1/a/2/1/stdout
/Ole