Distributing work to local and remote computers

[Eric Geoffroy]

I had a working command until I ran into files whose paths exceeded the maximum for the shell (or socket).

This works unless the paths exceed the max- 

cat '/Volumes/Cinera/SBO- Video/Python Videos/videofile_paths.txt' | parallel -q -S :,8/eric@serverA,8/eric@ServerB/eric@ServerC ffmpeg -i {} -vcodec libvpx -qmin 4 -qmax 10 -crf 6 -acodec libvorbis '/Volumes/Cinera/SBO- Video/Python Videos/Informit/9780134745954/media/video/pyfs_'{/.}.webm

Notes:

The files are on a shared drive mounted on local and remote computers. 

The files and directories have spaces in them. I use -q 

I send jobs to the remote computers in order of power

FFMPEG converts files into webm

I should be able to get— 8 cores * 4 computers = 32 simultaneous encodes. If I’m understanding correctly, the first 8 files go to the fastest host (local), the next 8 to serverA, and so on. 

And all was going well until the long file paths. I scoured the man page and examples and found two possibilities:

Plan B

this gem --workingdir.  I figured I would replace the "cat file" with sending the workingdir. 

To test this outside of ffmpeg I tried:

parallel -q --workdir '/Volumes/Cinera/SBO- Video/Python Videos 9780134745916/Safari/9780134745923/' -S 8/eric@10.105.241.211 file {} ::: '.mp4

output: 

*.mp4: cannot open `*.mp4' (No such file or directory)

But it worked when point to a file by name. 

parallel -q --workdir '/Volumes/Cinera/SBO- Video/Python Videos/videos/' -S 8/eric@ServerA file {} ::: 00_00_00.mp4

output: 

00_00_00.mp4: ISO Media, MP4 v2 [ISO 14496-14]

So then I tried just every imaginable variation to get a video list out of that directory. ls -1, find ., etc.

Plan C

In the Example-

Convert *.mp3 to *.ogg running one process per CPU core on local computer and server2:

parallel --trc {.}.ogg -S :,server2 'mpg321 -w - {} | oggenc -q0 - -o {.}.ogg' ::: *.mp3

The output file seems to come first. then the hosts. then the mpg321 command I don't grok. then that is piped to the ogg encoder with a duplicate output file. Why two outputs?

And now I'm stuck. Plan B or Plan C. Which is better? Where have I gone wrong?

thx

- Eric

GNU parallel 20170322