Re: only run submake if that submake needs to be run?

[Boris Kolpackov]

David Wuertele <address@hidden> writes:


>   define COMPONENT_template
>     $(shell make -s -C $(1) $(SUBMAKEARGS) report-installtargets): $(shell 
> make -s -C $(1) $(SUBMAKEARGS) report-sourcefiles); make -C $(1) 
> $(SUBMAKEARGS)
>   endef
> 
>   $(foreach component,$(SYSTEM_COMPONENTS),$(eval $(call 
> COMPONENT_template,$(component))))
> 
> The problem with this is that it is too coarse-grained:  the submake
> gets run if ANY of the source files are newer than ANY of the target
> files, even if those particular source files aren't used in the making
> of those particular target files.

It's also very slow. Look what you are doing here: for each sub-makefile
you run submake twice plus it goes via $(shell ). I believe it would be
way faster to just run sub-make once and let it do the job.


> If I could get the submake to output a set of dependencies, something
> like this:
> 
>   # make -C subdir123 tell-me-your-dependencies
>   /path/to/program1: ./subdir123/program1.c ./subdir123/program1-another.c ; 
> make -C subdir123
>   /path/to/program2: ./subdir123/program2.c ; make -C subdir123
>   /path/to/program3: ./subdir123/program3.c ./subdir123/program3-another.c ; 
> make -C subdir123
>   # make -C subdir456 tell-me-your-dependencies
>   /path/to/program4: ./subdir456/program4.c ./subdir456/program4-another.c ; 
> make -C subdir456
>   /path/to/program5: ./subdir456/program4.c ./subdir456/program5.c 
> ./subdir456/program6.c ; make -C subdir456
>   /path/to/program6: ./subdir456/program6.c ./subdir456/program6-another.c ; 
> make -C subdir456
>   # 
> 
> Then I could use that output to make a finer grained project makefile:
> 
>   SYSTEM_COMPONENTS := subdir123 subdir456
> 
>   define COMPONENT_template
>     $(shell make -C $(1) $(SUBMAKEARGS) tell-me-your-dependencies)
>   endef
> 
>   $(foreach component,$(SYSTEM_COMPONENTS),$(eval $(call 
> COMPONENT_template,$(component))))
> 
> This would then only run the submake if one of the origin files were
> newer.

I think you are trying to solve the wrong problem. Here is some background.
There are two ways to write makefiles for a project: recursive (your case)
and non-recursive. 

In recursive structure you in some sense aggregate sub-makefile graph 
into a single (unconditional) rule. The benefit is in makefiles which 
are usually quite simple and easy to maintain. The disadvantage is in 
the lose of precision: that aggregating rule is going to run 
unconditionally plus it usually hurts parallel builds badly.

In non-recursive structure make has precise information for the whole
graph in a single run. The benefits are obvious: no unnecessary 
commands are run and parallel builds scale very well. The disadvantage
is in writing and maintaining the makefiles; this could be very tricky
for anything but trivial projects.

Now let's see what you are trying to do. You have recursive build 
structure but you want to go an extra mile and partially dis-aggregate
those rules in the top makefile which will allow you to avoid 
unnecessary re-executions. Since the way you extract that information
is through $(shell ) calls to sub-makes I am pretty much sure it will 
be slower than to run normal recursive build.

Will it at least solve parallel build problem? The answer is no.
In fact you will probably render parallel builds completely unusable.
Consider, for example, something like this:

# make -C subdir123 tell-me-your-dependencies

/path/to/program1: ./subdir123/program1.c ./subdir123/common.c
        make -C subdir123 /path/to/program1

/path/to/program2: ./subdir123/program2.c ./subdir123/common.c
        make -C subdir123 /path/to/program2

Imagine now we run `make -j2'. Can you say for sure what's going 
to happen to `./subdir123/common.o'?


>From this you can derive one of the golden rules of make: 

"Have complete dependency graph or run it all the time." ;-)

> Is there any way to achieve this goal with GNU Make 3.80?

If you are still not convinced this is not the way to go, check out -d 
and -p flags for GNU make.


-boris