Re: Parallel input from growing file?

[xmoon 2000]

On 15 December 2014 at 09:06, xmoon 2000 <xmoon2000@googlemail.com> wrote:
> On 14 December 2014 at 14:08, Hans Schou <chlor@schou.dk> wrote:
>> Start reading this chapter in the man page:
>>   EXAMPLE: GNU Parallel as queue system/batch manager
>>
>> You are not the first who (mis-)use Gnu Parallel as a queue manager.
>>
>> /hans
>>
>> 2014-12-14 12:01 GMT+01:00 xmoon 2000 <xmoon2000@googlemail.com>:
>>>
>>> Hi,
>>>
>>> Currently I read jobs from a file on my c drive that gets written to
>>> by other scripts, and my master script runs each job found in
>>> background with &. I want to use parallel instead of my own script.
>>>
>>> I would like parallel to monitor this file and run any jobs in it with
>>> as many cores as are available, or wait until new jobs are appended to
>>> the file. Running 24/7.
>>>
>>> What is the syntax for doing this?
>>>
>>> My jobFile looks like this:
>>>
>>>
>>> /myApp prm1 parm2 parm2
>>> /myApp parm1 parm2
>>> /myApp parm1 parm2 parm3 parm4
>>>
>>> I could do:
>>>
>>> tail -f jobFile  | parallel
>>>
>>> Is this the best approach?
>>>
>>
>
>
> Hi Hans,
>
> Yes I had read that section before. However, I assume this is a common
> request and wondered if any other functionality had been created that
> would allow a jobQueue to be consumed? i.e. the jobs in the jobQueue
> are deleted as they are processed?
>
> Also a run "run up to x jobs starting now - would be very useful"


Any thoughts on this Ole?