Re: Undesirable overhead in GNU Make 4.2.1.

[Reinier Post]

On Tue Feb  6 15:12:45 2018, address@hidden (Hans Ottevanger) wrote:

>  Hi,
> 
> We are attempting to use GNU Make 4.2.1 to build a large software tree
> (~290000 targets).
> 
> The build hosts we are testing on are (virtualized) quad core X86-64
> machines running RHEL 6. Our code tree is currently on "locally
> perceived" storage. We compiled GNU Make 4.2.1 from sources.
> 
> Our main concern at this moment is the time needed for "nil
> incrementals", i.e. the time needed to establish that nothing has
> changed, so nothing needs to be done. If we put all dependencies in
> the Makefile itself or in one single included makefile, we manage to
> do a nil incremental in about 12 seconds.
> 
> But we plan to use auto-generated dependencies and thus include a .d
> file in our main Makefile for each target. Currently auto dependency
> generation is possible for about 90000 of our targets. Nil incremental
> builds cost increases to about 140 seconds in that case, which is
> quite unacceptable for our usage. We found that this overhead
> increases approximately quadratically with the number of included
> dependency files.
> 
> It appears that this behavior is caused by the need to maintain the
> variable MAKEFILE_LIST. The names of all included makefiles are
> appended one by one to the ever growing string representing this
> variable.This also explains why memcpy() and strlen() champion the
> profiled runs we did and also the quadratic behavior we observed. When
> we change the source (read.c) so that the MAKEFILE_LIST is not
> maintained at all, the overhead is gone and we are back to about 12
> seconds again.
> 
> Are there any ideas (or opinions) about how to get rid of this type of
> undesirable overhead? One of our own ideas is to provide a special
> variant of -include (called e.g. dinclude?), intended to include just
> .d files, without maintaining MAKEFILE_LIST. On the other hand it
> might be feasible to improve the efficiency of appending to a
> variable. Any suggestions?

The latter should be possible by accumulating the elements in a
hash table or linked list and composing the resulting string
afterwards, allocating it once and strcpying the elements
into position.

Meanwhile, I think having a $MAKEFILE_LIST value of over a million
characters will cause problems when you try to use it. So it does
seem to be a good idea to add some kind of option for curbing or
preventing its creation.

-- 
Reinier Post
TU Eindhoven