Re: Transferring files while jobs are running

[Ole Tange]

On Sun, Nov 28, 2010 at 9:49 AM, R. Tyler Croy <tyler@monkeypox.org> wrote:
> As I mentioned in my previous email to the list, I'm encoding video files this
> evening, which means transferring files hundreds of megabytes large across my
> relatively slow (100Mb/s) home network.
>
> The encoding of a particular video file can take anywhere from 10-45 minutes
> depending on the machine, is there a means of starting the transfer of the 
> file
> for the "next job" on a given machine while it's processing the previous job?

No. The reason is that we do not know on which server the next job
will be run until a job actually finishes.

But you can start a job more than the system has CPUs by using -j+1.

sem can help you make sure you only run the compute part when there is
a spare cpu:

cat filelist | parallel -j+1 --trc {.}.ogg -S .. sem -j+0
encodingscript {} -o {.}.ogg

This will run 9 jobs on a 8 core machine but sem will make sure
encodingscript will only run 8 times in parallel on the machine. So 9
files will be transferred, then 8 encodings will be started, the last
one being blocked by sem until the first of the 8 encodings finishes.
Then yet another file will be transferred.

(sem is part of GNU Parallel).


/Ole