Re: Out of memory when doing --pipe --block ?

[hubert depesz lubaczewski]

On Thu, Feb 08, 2018 at 07:50:22PM +0100, Ole Tange wrote:
> In practice this means you need 2.3 GB of free RAM and 3.8 GB of free
> virtual memory (RAM+swap) to run this.
> 
> Given your output from 'free' I think that if you add 2 GB of swap,
> then that will solve your issue. Just because Perl loves swap, it does
> not mean that it will start swapping - it just wants to feel safe that
> it could if it wanted to. I have seen the same behaviour of other
> programs, so my guess is that you might get more performance from your
> system simply by adding a little swap space.
> 
> The other option is to use --cat as described on
> http://lists.gnu.org/archive/html/parallel/2018-01/msg00027.html

Yes, but --cat creates temp files, which I can't have as it slows down
things considerably.

Anyway - looks like I simply can't use parallel. It's memory
requirements are (from my perspective) impossible to handle, and I still
don't understand what exactly is the benefit of buffering stuff in
memory.

depesz