Re: Out of memory when doing --pipe --block ?

[Ole Tange]

On Wed, Feb 7, 2018 at 3:42 PM, hubert depesz lubaczewski
<depesz@depesz.com> wrote:

> I'm trying to work with parallel to split tarball, and then
> compress/encrypt/upload-to-s3, but it's failing me.
>
> I'm using parallel 20180122 on 64 bit linux.
>
> My mem looks like:
>              total       used       free     shared    buffers     cached
> Mem:       4046856    2483552    1563304    1355128       5488    2111912
> -/+ buffers/cache:     366152    3680704
> Swap:            0          0          0
>
> Not very beefy, as it's just one small test virtual server.
>
> data to be split is ~ 33G.
>
> My command is, more or less:
> tar cf - /var/lib/postgresql | parallel -j 5 --pipe --block 360M --recend '' 
> handle_single_part /tmp/test.pass us-east-1 "${s3_tarball_part}"

Your issue is very similar to
http://lists.gnu.org/archive/html/parallel/2018-01/msg00020.html

If I run:

  yes "`seq 1000`"| parallel -j 5 --pipe --block 360M --recend '' sleep 1000

I get:

$ ps aux | grep perl | field 5 | summer
3869324

$ ps aux | grep perl | field 6 | summer
2319424

So the virtual size is 3.8 GB and the resident size is 2.3 GB.

In practice this means you need 2.3 GB of free RAM and 3.8 GB of free
virtual memory (RAM+swap) to run this.

Given your output from 'free' I think that if you add 2 GB of swap,
then that will solve your issue. Just because Perl loves swap, it does
not mean that it will start swapping - it just wants to feel safe that
it could if it wanted to. I have seen the same behaviour of other
programs, so my guess is that you might get more performance from your
system simply by adding a little swap space.

The other option is to use --cat as described on
http://lists.gnu.org/archive/html/parallel/2018-01/msg00027.html


/Ole