bug-parallel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

GNU Parallel Bug Reports A suggestion: --shuf and -k


From: paralleluser
Subject: GNU Parallel Bug Reports A suggestion: --shuf and -k
Date: Fri, 30 Jun 2017 17:11:12 -0400

Friends

A suggestion that merits your comments and review:

--shuf does exactly what the man page says it does, but when you combine --shuf 
and -k, the -k does nothing, --shuf rules over -k

I'm going to propose that combining --shuf and -k that this happens:

        the jobs are still processed randomly
        but the output be in order as the true input
        
When do you use this?  Assume your input to parallel process is:

        server1/resource1
        server1/resource2
        server1/resource3
        ...etc...
        server2/resource1
        server2/resource2
        ...etc...
        ...up to server50
                
For human processing reasons, it is easier to keep all the server/resource 
input lines in ASCII sort order

For computer processing reasons, server 1 is going to hate you if you are 
hitting it with a lot of requests all at the same time

Thus with the "--shuf -k" combo, the sever loads will be spread around, but you 
will get your data back in the same order.

Comments welcome........thanks



reply via email to

[Prev in Thread] Current Thread [Next in Thread]