If I may ask while we're at it: can we make some option to have GNU Parallel NOT buffer things and/or not output things? Some operations are heavy on I/O and not computationally intensive, so there's no need for either buffering or output from them :)
> Dnia 9 luty 2018 o 09:45 hubert depesz lubaczewski <depesz@depesz.com> napisał(a): > > > On Thu, Feb 08, 2018 at 07:50:22PM +0100, Ole Tange wrote: > > In practice this means you need 2.3 GB of free RAM and 3.8 GB of free > > virtual memory (RAM+swap) to run this. > > > > Given your output from 'free' I think that if you add 2 GB of swap, > > then that will solve your issue. Just because Perl loves swap, it does > > not mean that it will start swapping - it just wants to feel safe that > > it could if it wanted to. I have seen the same behaviour of other > > programs, so my guess is that you might get more performance from your > > system simply by adding a little swap space. > > > > The other option is to use --cat as described on > > http://lists.gnu.org/archive/html/parallel/2018-01/msg00027.html > > Yes, but --cat creates temp files, which I can't have as it slows down > things considerably. > > Anyway - looks like I simply can't use parallel. It's memory > requirements are (from my perspective) impossible to handle, and I still > don't understand what exactly is the benefit of buffering stuff in > memory. > > depesz >