[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Out of memory when doing --pipe --block ?
From: |
Ole Tange |
Subject: |
Re: Out of memory when doing --pipe --block ? |
Date: |
Thu, 8 Feb 2018 19:50:22 +0100 |
On Wed, Feb 7, 2018 at 3:42 PM, hubert depesz lubaczewski
<depesz@depesz.com> wrote:
> I'm trying to work with parallel to split tarball, and then
> compress/encrypt/upload-to-s3, but it's failing me.
>
> I'm using parallel 20180122 on 64 bit linux.
>
> My mem looks like:
> total used free shared buffers cached
> Mem: 4046856 2483552 1563304 1355128 5488 2111912
> -/+ buffers/cache: 366152 3680704
> Swap: 0 0 0
>
> Not very beefy, as it's just one small test virtual server.
>
> data to be split is ~ 33G.
>
> My command is, more or less:
> tar cf - /var/lib/postgresql | parallel -j 5 --pipe --block 360M --recend ''
> handle_single_part /tmp/test.pass us-east-1 "${s3_tarball_part}"
Your issue is very similar to
http://lists.gnu.org/archive/html/parallel/2018-01/msg00020.html
If I run:
yes "`seq 1000`"| parallel -j 5 --pipe --block 360M --recend '' sleep 1000
I get:
$ ps aux | grep perl | field 5 | summer
3869324
$ ps aux | grep perl | field 6 | summer
2319424
So the virtual size is 3.8 GB and the resident size is 2.3 GB.
In practice this means you need 2.3 GB of free RAM and 3.8 GB of free
virtual memory (RAM+swap) to run this.
Given your output from 'free' I think that if you add 2 GB of swap,
then that will solve your issue. Just because Perl loves swap, it does
not mean that it will start swapping - it just wants to feel safe that
it could if it wanted to. I have seen the same behaviour of other
programs, so my guess is that you might get more performance from your
system simply by adding a little swap space.
The other option is to use --cat as described on
http://lists.gnu.org/archive/html/parallel/2018-01/msg00027.html
/Ole