Re: Unexpected positional replacement strings substitution

[aero]

"cat SERVERLIST.txt | parallel --pipe -N1 -S worker1,worker2 --roundrobin..." makes forked ssh connection per every "--pipe -N1 --roundrobin" dispatch.

<TEST>
list.txt contains 1~10

$ cat list.txt | parallel --pipe -N1 --roundrobin -S worker1 --cat 'echo {}'
/tmp/parEkNFp
/tmp/parTOLj5
/tmp/parOpT9c
/tmp/parBR2ug
/tmp/parx0wSN
/tmp/pareOwLq
/tmp/par3hhaU
/tmp/parvAt7l
/tmp/parUtMFo
/tmp/parM4nHL

It makes 10 connections and makes tmp file 10 times which contains 1 line.

So Forking load of master server is not reduced.

If I have 4000 target servers, 1 master and 4 worker servers.

What I want is...

* master -> worker1~4 -> target 1~4000

- Master has only 4 ssh control connection to workers.( I though that --controlmaster option will work like this )

- Master send divided all_list/4 target list to workers.

- Each workers runs 1000 tasks(connect to target and execute some script or command which takes few seconds to finish.)

All commands you showed me need master's new proccess fork and ssh connection(session) to worker per every task on worker server.
 

 

On Fri, Feb 24, 2017 at 6:02 AM, Ole Tange <ole@tange.dk> wrote:

On Thu, Feb 23, 2017 at 6:17 AM, aero <chahkang@gmail.com> wrote:

> I am going to try like the following.
> Send divided list to workers. And workers runs own parallel against the list
> from master.
>
> cat SERVERLIST.txt | parallel --pipe -N[ int(lines count of SERVERLIST.txt/
> number of workers) ] -S worker1,worker2,... 'cat | parallel ...'
>
> Is there any other good way to achieve job like this ??

cmd='echo "joe  two  space"'
cat SERVERLIST.txt | parallel --pipe -N1 -S worker1,worker2
--roundrobin --cat parallel --slf {} --nonall -q "$cmd"

/Ole