Re: Performance loss when trying to use grouped targets

[Stephen Touset]

On Mar 4, 2024 at 18:01:14, Dmitry Goncharov <dgoncharov@users.sf.net>
wrote:



>
>
>
>
>
>    On Mon, Mar 4, 2024 at 2:27 PM Stephen Touset via Users list for the
> GNU implementation of make <help-make@gnu.org> wrote:
>
> Is there a way to write a rule that expresses something closer to
>
> “each file in the build directory depends upon its corresponding
>
> file in the source directory” but still uses grouped targets?
>
>
> A grouped targets rule like
> a.x b.x &: a.q b.q; cp a.q b.q build/
> tells make that a.x depends on a.q and b.q and b.x depends on a.q and
> b.q. Which is not "each file depends on its corresponding file".
>
> The above grouped rule is the same as
> a.x: a.q b.q; cp a.q b.q build/
> b.x: a.q b.q; cp a.q b.q build/
>
> This rule tells make to compare mtimes of a.x against those of a.q and
> b.q, rather than against mtime of a.q alone.
>

Right, I note as much in my original email.

If you want to copy only the changed file alone.
> Then
> a.x: a.q; cp $< build/
> b.x: b.q; cp $< build/
>

My entire goal here is to improve build performance by avoiding individual
calls to cp because a single invocation is significantly faster. This is a
project with thousands of files that need to be copied into the build
directory. A rough benchmark:

❯ time find src -type f -exec cp -a {} build \;

________________________________________________________
Executed in   26.32 secs    fish           external
   usr time    1.77 secs    1.11 millis    1.77 secs
   sys time    9.63 secs   22.94 millis    9.61 secs


❯ time cp -a src build

________________________________________________________
Executed in    1.69 secs    fish           external
   usr time    0.06 secs    0.86 millis    0.06 secs
   sys time    1.56 secs   19.67 millis    1.54 secs


I’d like to have my cake and eat it too, if possible. I’m hoping there’s a
way to express

build/aaaa.x: src/aaaa.q; cp $< $@
build/aaab.x: src/aaab.q; cp $< $@
...
build/zzzz.x: src/zzzz.q; cp $< $@


but with the semantics (and performance in this case) of a grouped target.
As you note,

build/aaaa.x build/aaab.x ... build/zzzz.x: src/aaaa.x src/aaab.x ...
src/zzzz.x      cp -r src build


doesn’t correctly express the semantics since each file in the build
directory will depend upon every file in the src directory. This does have
the desired effect of a single cp, but at the cost of an explosion in make
analysis time.

I’d love something that combines the best of both worlds, and lets me
express that each file in build depends only on its counterpart in src, but
that all targets can be made by one invocation of the rule. Something
(hypothetically) like:

aaaa.x&: aaaa.qaaab.x&: aaab.q...zzzz.x&: zzzz.q        cp -a src build