Re: ulimit memory to 1/X'th of physical memory for jobs

[Ole Tange]

On Thu, Aug 9, 2012 at 3:24 PM, Rhys Ulerich <rhys.ulerich@gmail.com> wrote:

>> Would it make sense to have a setting in GNU
>> Parallel that automatically run 'ulimit ' with the relevant amount of
>> memory, so if you ask for X jobs to be run on a given server, then
>> each job is only allowed 1/X'th of the memory on the machine.
>
> I like it.  Maybe '--whatever 100%' is up to the limit of the machine
> and '--whatever 100M' allows setting a specific size to be divided
> amongst the jobs.

Interesting idea.

> I wondered if one could set a memory limit on a group so that the on
> average the processes could use no more than 1/Xth of the memory but
> any individual process might employ more than 1/X.  Reading the ulimit
> description in the bash(1) man page, I noticed this ulimit option:
>
>    -v     The maximum amount of virtual memory available to the shell
> and, on some systems, to its children
>
> It seems the goal you have is to enforce that none of the processes
> swap.  Do you think a 'ulimit -v'-like behavior could be collectively
> attached to the group or processes that GNU parallel spawns?  If so,
> could the virtual limit be set to be only the physical memory?

Doing it per process would simply require a 'ulimit -v/-m' to be run
as part of the command. This will work on remote machines, too.

Doing it for the group of jobs will not work remotely, as each remote
jobs is started through its own ssh. To run locally it requires
changing the ulimit from inside perl. BSD::Resource has setrlimit
which seems to do what we want. But I really do not want to require
people to install yet another Perl module - especially not to get this
fairly limited functionality.

So I think the 'ulimit -v/-m' run as a wrapper of the command is
currently the only realistic.

- o -

Should it take an argument? I tend to say no. The normal situation
will be that you want a safety net that will save you if one of the
jobs takes more the a 1/X'th of the RAM, where X is the number of jobs
running on that machine; thus it should be computed on the machine
running the jobs. And in the odd case that you do not want that, you
can run 'ulimit' manually.

What would be a reasonable name? --memlimit? --ramlimit? --maxram?
--swap-safety-net?

>> For most situations it will not make a difference, so I am interested
>> in whether you will anticipate more surprise by having your big job
>> killed or the happyness by not swapping the machine to death?
>
> The principle of least surprise says running too many memory hungry
> processes should swap the machine into the ground, IMHO.

After giving it some thoughts I tend to agree: The safety net should
not be default.


/Ole