Re: Parallel input from growing file?

[Hans Schou]

2014-12-17 12:23 GMT+01:00 Ole Tange <ole@tange.dk>:

On Wed, Dec 17, 2014 at 12:01 PM, xmoon 2000 <xmoon2000@googlemail.com> wrote:
> On 15 December 2014 at 09:06, xmoon 2000 <xmoon2000@googlemail.com> wrote:
>> On 14 December 2014 at 14:08, Hans Schou <chlor@schou.dk> wrote:

>> Yes I had read that section before. However, I assume this is a common
>> request

I think it has been discussed 2-3 times on the mailinglist for the
whole lifespan of GNU Parallel. So either very few are using it or
they are happy with the current functionality

 

(or they are so
disgruntled, that they changed to a real queueing system).

It should be raily simple with RabbitMQ

https://www.rabbitmq.com/tutorials/tutorial-two-python.html

#1 If you still want to use GNU Parallel

#2 To only use a queing system

/hans

>> and wondered if any other functionality had been created that
>> would allow a jobQueue to be consumed? i.e. the jobs in the jobQueue
>> are deleted as they are processed?

GNU Parallel was never intended as a full blown queue system, so if
you "abuse" it for that, then I think it is not too much to ask that
if you feel the queuefile is too big, that you shut down GNU Parallel
and delete the processed jobs manually.

A tip is use use -E to make GNU Parallel stop at a magic value. Then
remove all lines up to the magic value and go again:

tail -f jobqueue | parallel -E StOpHeRe
perl -e 'while(<>){/StOpHeRe/ and last};print <>' jobqueue > j2
mv j2 jobqueue

>> Also a run "run up to x jobs starting now - would be very useful"

You have to explain how this is not what GNU Parallel already does:

# Run 1 jobs per core (during the night)
echo -j100% > jobsfile
true >jobqueue; tail -f jobqueue | parallel -j jobsfile -S .. &
sleep 36000
# Run 0.5 jobs per core (during the day) - starting from next complete job
echo -j50% > jobsfile


/Ole