parallel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Dividing input among the jobs to run when using -X or -m


From: Ole Tange
Subject: Dividing input among the jobs to run when using -X or -m
Date: Fri, 12 Nov 2010 15:23:37 +0100

I just read: http://web.archiveorange.com/archive/v/LXyZOZB96u81jByJuXoS

  parallel -j +0 -m command args ::: *

  where "command args" is your command and options;
  the "-j +0 -m" divides all the file names across one instantiation per
  processor core.

That is not the case though. GNU Parallel cannot do that at the moment.

I have been thinking about implementing something like that, but there
is a drawback. If the input is taken from a pipe, then we need to read
until the pipe is empty or until we have enough arguments to fill up a
command line for each core. This will affect a situation like:

find /dir/with/lots/of/files/and/dirs | parallel -j+0 -m dostuff

Currently we start dostuff as soon as we have the first full line.
With the idea above we would have to wait until we have a full line
for each core or the pipe is empty.

The drawback does not apply if reading from a file or from the command line.

So my question is: Would this feature help you in general? Or would it
annoy you because of the drawback?


/Ole



reply via email to

[Prev in Thread] Current Thread [Next in Thread]