Re: How do parallel builds scale?

[Ralf Wildenhues]

Hi Ludo,

* Ludovic Courtès wrote on Thu, Mar 03, 2011 at 04:42:52PM CET:
> I ran a series of build time measurements on a 32-core machine, with
> make -jX, with X in [1..32], and the results are available at:
> 
>   http://hubble.gforge.inria.fr/parallel-builds.html

Thank you!  Would you be so kind and also describe what we see in the
graphs?  I'm sorry but I fail to understand what they are showing, what
the axes really mean, and how to interpret the results.

> There are packages whose configuration phase is noticeably longer than
> the build time.

Yes, we knew that.  Can you please also mention whether you used a
config.site file?  Since using a config.cache file for one-time builds
is not relevant, I'm assuming that is not necessary to know.  But it
would be fairly cool to know how development could be sped up.  E.g.,
one thing you could try is, after configure -C once, save the
config.cache file somewhere, remove the build directory, rerun configure
with CONFIG_SITE pointing to that moved cached file.  That could give a
more realistic impression of how expensive configure overhead is while
developing.  (I understand that that isn't so interesting for a
distribution.)

Packages with many configure scripts (like GCC, binutils) will not
benefit in the same way, of course.

> OTOH, packages with a long build time (with a sequential time in
> boldface) don’t scale well either, but that is not configure’s fault.

I suppose several packages' check bits would benefit from Automake's
parallel-tests feature.  A few of the packages (using an Autotest test
suite: Autoconf, Bison) would benefit from you passing
TESTSUITEFLAGS=-jN to make.

FWIW, parallelizability of Automake's own 'make check' has been improved
in the git tree (or so at least I hope).

I am fairly surprised GCC build times scaled so little.  IIRC I've seen
way higher numbers.  Is you I/O hardware adequate?  Did you use only -j
or also -l for the per-package times (I would recommend to not use -l).

Thanks,
Ralf