[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Dividing input among the jobs to run when using -X or -m
From: |
Ole Tange |
Subject: |
Dividing input among the jobs to run when using -X or -m |
Date: |
Fri, 12 Nov 2010 15:23:37 +0100 |
I just read: http://web.archiveorange.com/archive/v/LXyZOZB96u81jByJuXoS
parallel -j +0 -m command args ::: *
where "command args" is your command and options;
the "-j +0 -m" divides all the file names across one instantiation per
processor core.
That is not the case though. GNU Parallel cannot do that at the moment.
I have been thinking about implementing something like that, but there
is a drawback. If the input is taken from a pipe, then we need to read
until the pipe is empty or until we have enough arguments to fill up a
command line for each core. This will affect a situation like:
find /dir/with/lots/of/files/and/dirs | parallel -j+0 -m dostuff
Currently we start dostuff as soon as we have the first full line.
With the idea above we would have to wait until we have a full line
for each core or the pipe is empty.
The drawback does not apply if reading from a file or from the command line.
So my question is: Would this feature help you in general? Or would it
annoy you because of the drawback?
/Ole
[Prev in Thread] |
Current Thread |
[Next in Thread] |
- Dividing input among the jobs to run when using -X or -m,
Ole Tange <=