Persistent queue system with remote execution using parallel?

[Åsmund Skjæveland]

Hello,
Is it possible to use GNU Parallel to implement a persistent queue system with 
remote execution?

My setup is: two compute servers, identical configuration, no shared network 
drive. The job in question downloads its input data over FTP, and --return is 
used for the processed data. 

Using 'parallel -S 4/server1 -S 4/server2 command' works fine as long as I only 
have one set of tasks at a time. However I'd like to run both hourly jobs and 
daily jobs.
If I just run them as two separate simultaneous Parallel jobs the compute 
servers will run out of memory and swap themselves to a snail's pace.  So I'd 
like to run both sets of jobs in a shared queue.

Will a named semaphore do what I want, if I use the same semaphore name for 
both ses of jobs?

Will it use the correct number of CPUs (4) on each server if I specify '-S 
4/server1 -S 4/server2', or do I also need to specify --jobs?


The hourly job takes around 15-20 minutes to complete. Ideally I'd want the 
hourly job to have higher priority so that the daily jobs are only running when 
there are no hourly jobs queued, but it seems this is not possible with 
Parallel.




Åsmund Steen Skjæveland, PhD
Senior Engineer
Norwegian Mapping Authority, Geodetic Insititute