Re: Catch OOM kills

[Ole Tange]

On Fri, May 4, 2018 at 7:23 PM, Douglas Pagani
<redstonefreedom@gmail.com> wrote:
> On Fri, May 4, 2018 at 8:42 AM, John <johnanders@posteo.de> wrote:
>>
>> How can I catch if the program I have called with parallel gets killed by 
>> the kernel due to memory space.
:
>> I would like to have an option that returns me all the jobs that were not 
>> able to be finished. Is this possible?

> You can use parallel --joblog ~/my.log to output several pieces of 
> information about jobs. One of those pieces is "ExitVal", which will tell you 
> not only that your job completed unsuccessfully, but with what exit code. For 
> example, instead of having to check dmesg for a "Out of memory: Kill process 
> ..." message, you can safely assume 143 is from linux's OOM killer having 
> sent your process a SIGTERM (128 + 15).
>
> I usually run an ad hoc script to pick up the "stragglers" after a larger 
> run, by parsing that file for any non-zero ExitVal's, and re-invoking the 
> full command line associated with it. Of course, if the exit code was due to 
> something deterministic, you'll just get non-zeros again and again, without 
> first fixing the problem with the data/args of those specific invocations 
> first.

I would reckon that is a good approach. If your jobs have very varying
memory usage, then first run a lot of them in parallel:

  parallel --joblog my.log -j100% [...]

When that is done run all failed jobs again, but run only a single job
at a time to give it the most memory available:

  parallel --retry-failed --joblog my.log -j1

or:

  parallel --resume-failed --joblog my.log -j1 [...]

This last part is basically Douglas' ad hoc script.

The difference between --retry-failed and --resume-failed is described
in the man page.


/Ole