Re: Limiting memory used by parallel?

[hubert depesz lubaczewski]

On Thu, Jan 25, 2018 at 11:26:24AM -0500, Joe Sapp wrote:
> Hi depesz,
> 
> On Thu, Jan 25, 2018 at 10:33 AM, hubert depesz lubaczewski
> <depesz@depesz.com> wrote:
> [snip]
> > But it looks that parallel itself is consumming HUGE amount of memory
> > - comparable with size of /some/directory itself.
> >
> > Server that I run it on has 64GB of ram, and the script gets killed
> > after ~ 3 minutes with "Out of memory!" error.
> 
> Have you tried using --memfree?  That will limit parallel from
> spawning new processes

OK, I read the docs, and I don't get it.

Why would I want to limit parallel from spawning new processes?

The problem, in my opinion, is that parallel is using memory for some
kind of buffer when splitting data, instead of operating in some small
block size, like 8kb, or whatever to copy data form input to one of the
outputs.

The final scripts do not use lots of ram. It's just parallel that is
using over 10GB of ram to split data and put it into multiple pipes.
This doesn't look OK to me - isn't it a bug?

Best regards,

depesz