Re: Cannot use more than 16 remote servers

[Ole Tange]

On Sat, Jul 13, 2013 at 7:43 PM, Nick Boutelier <nboutelier@gmail.com> wrote:

> Hi, in reference to this stackoverflow post, im emailing you the output of
> my jobs.
>
> http://stackoverflow.com/questions/16348202/using-maximum-remote-servers
>
> I have a 16 core local ubuntu machine and am trying to send jobs out to 17
> machines. The jobs only get sent to 16 though.

I just tested if the number of cores on the starting host has any
effect on the number of hosts to run on:

# setup
seq 255 | parallel -j10 echo 127.0.0.{} >~/.parallel/255hosts
seq 255 | parallel -j10 ssh -oStrictHostKeyChecking=no 127.0.0.{} true
# run
seq 255 | parallel -j1 -u --slf ~/.parallel/255hosts 'echo {} in;sleep
100;echo {} out'

On my machine that gives the first 255 ins then a waiting time and
finally the 255 outs. This shows that GNU Parallel happily spawns 255
jobs on 255 different hosts.

So something else is going on in your case.

The output's 3rd to last line says the reason:

    Not starting: no more file handles

So you need more filehandles if you want more jobs (Look at ulimit -n).

Next version will print a warning instead of just putting the
information in the debug output.

As a work around you can use --ungroup. This, however, will cause the
output to get mixed, but if you do not care about the output, then
that might just work for you.


/Ole