Re: reporting failed jobs at the end of a gnu parallel run

[Ole Tange]

On Thu, Apr 7, 2011 at 12:06 PM, Jesse Alama <jesse.alama@gmail.com> wrote:
> Hi Ole,

If you use the mailinglist, then others can help you as well.

> Perhaps I'm missing something in the GNU parallel documentation, but I cna't
> seem to find what I'm looking for.  My problem is that I'd like to get a
> list of all the jobs I give to parallel that failed.  I often use --eta, and
> if a job fails it might give some output, and then the ETA display gets
> garbled, then the ETA gets reset, and the error line (or lines) get shifted
> upward.

ETA is crap. It is already on my todo to get that better.

>  If there are a lot of errors, then I can't always econstruct just
> what jobs failed.  It would be great if I could do something like
>
>  parallel --report [other options] [command]

Maybe:

parallel --joblog /tmp/report.txt my_cmd

Or:

parallel "my_cmd {} 2>{}.error >{}.out"

Or:

parallel "echo the args: {} > \$PARALLEL_SEQ.log; my_cmd {} 2>
\$PARALLEL_SEQ.log; echo \$? > \$PARALLEL_SEQ.log"

> and then get a summary at the end:
>
> Job  Exit Code Error Output
> job1 0         (none)
> job2 1         Can't open directory /foo/bar
> job3 0         (none)
> job4 2         Disk full

I know how get the exit code, and I will make that part of the
--joblog in next release. But what is the "Error Output"? Is that the
output on stderr? In that case it can be multiple lines, so it will
not work for a report.

> This would help me enormously.  For some reason, some jobs that I run seem
> to fail mysteriously when I run them under GNU parallel.  For example, I try
> to apply an XSLT stylesheet to tens or hundreds of thousands of XML files,
> and sometimes I get strange results (e.g., applying the stylesheet results
> in an empty XML file, or the path of the target XML file gets mangled).  I
> doubt that this is GNU parallel's fault; my guess is that something is going
> wrong when my disk is trying to write so many things concurrently, or
> perhaps it is my shell (maybe something weird is happening when it tries to
> construct a command line with hundreds of thousands of arguments?).
>
> In any case, this is something for further investigation, and I can write to
> you separately about this.  My request for a summary at the end of a run
> stands alone.

Look at --joblog.

Also you might want to look at --halt-on-error if you want parallel to
stop when the first command fails. Combined with -v that might be what
you are looking for.

> Best,
>
> Jesse
>
> --
> Jesse Alama
> http://centria.di.fct.unl.pt/~alama/

/Ole