Re: [help-gnubatch] ( [PATCH] Bump version in debian/changelog to 1.4 also ) + newbie question

[Peter Valdemar Mørch]

Thanks for replying!

On Wed, Jun 6, 2012 at 12:52 PM, Reuti <address@hidden> wrote:
> You could put a sleep with the necessary amount of seconds in the jobscript 
> and start it at the full minute before, so that you can start it at any 
> second.

:-( I was affraid of that. That also means, I take it, that all the
details of the scheduling - when to run what - is left up to me/us.

> Is it to run in a cluster of machines or only on a local one? To me it sounds 
> like you need some features from GridEngine (like the accounting which you 
> can access later), and on the other hand some kind of "real-time-feature" 
> from GNUbatch.

No, it is to run all 1200 mostly tiny, light-weight jobs on one
machine, just like nagios does. Typically ping, check_tcp which
basically only tries to connect to a TCP port. But could be anything.

Didn't know about GridEngine, will look into that. I'm just getting
worried that the scheduler's overhead is much much higher than the
jobs' with these many tiny jobs!

What we're currently doing is starting a process every 15 min, that
runs N child processes at a time in parallel until all 1200 are done.
The consequence of that is we get high-ish load around 0:15, 0:30,
0:45 etc. and would love to use "a scheduler" to smooth out that load,
and hopefully use existing tool infrastructure to get more debugging
insight (execution times, output etc.) and runtime control.

> Do you have more machines than jobs, i.e. all 1200 jobs should run at the 
> same time on a bunch of machines?

One machine. Cool if it lends itself to more machines, but this is
handled fine by one machine with our brute force N parallel approach.
The trick is picking N since execution times vary. And then I thought:
"Somebody must have tackled this previously and created an open source
project!" :-)

Tschüss,

Peter
-- 
Peter Valdemar Mørch
http://www.morch.com