Re: Parallel make

[Paul Smith]

On Wed, 2015-04-29 at 13:50 -0600, Ryan P. Steele wrote:
> The multithreaded version of make (-j#) is wonderful, and we have made
> great use of it.  Because we're dealing with some very large code, 
> however, it would be great to be able to parallelize compilation over 
> multiple machines.  I can't seem to find any option for doing this, 
> though.  Does such functionality exist?  It would appear to be a
> fairly straightforward extension of multithreading, especially on a
> network file system, but thus far, we haven't been able to make it
> work.  Any help would be appreciated.

Just to be clear, GNU make is not multithreaded.  It simply spawns
multiple processes and lets them run in parallel.

GNU make has no built-in capability to use multiple machines:
conceptually it may be a straightforward extension but the effort needed
to communicate between multiple systems over a network, send and receive
results reliably, kill jobs when someone stops the main make process
with ^C, etc. is FAR out of GNU make's current wheelhouse.

You can get a very cheap implementation, if you're willing to live with
many prerequisite configuration requirements such as a networked
filesystem, SSH access that doesn't require a password, etc. by writing
your own script to forward the job via SSH and setting the GNU make
SHELL variable to point to your script.

Luckily, if you are building C or C++ code someone has already done all
the necessary work for you.  I recommend you investigate the distcc
package: https://code.google.com/p/distcc/

Cheers!