Re: [Automake-NG] [PATCH 00/11] Several changes to parallel-tests support

[Stefano Lattarini]

> On 05/09/2012 06:03 PM, Akim Demaille wrote:
>
> Deeply nested Make macros can be extremely costly, especially
> because they are not lazy, this is really call by name: when used
> several times, they are evaluated several times.  There must be
> some good reason for this, but then result is longer compilation,
> which is _bad_.
>
> So I think you should at least try to make some measurements.
>
Ouch, you were quite right :-(

I've tried with a dummy testsuite having 5000 dummy tests (all passing);
skipping the measurements for "make check" (which would be worthless,
since 99% of the time is spent in the test driver script), here are the
numbers for other relevant targets (on my slowish desktop):

===================================
 BEFORE the series is been applied
===================================

$ make all                                      # 12 times
1.5 seconds

$ make recheck                                  # 1 time
2.6 seconds

$ make recheck TESTS='1 2 3 4 5 6 7 8 9'        # 6 times
1.2 seconds

$ make check RECHECK_LOGS=                      # 1 time
4 seconds

===================================
 AFTER the series has been applied
===================================

$ make all                                      # 12 times
10 seconds

$ make recheck                                  # 1 time
5.5 seconds

$ make recheck TESTS='1 2 3 4 5 6 7 8 9'        # 6 times
3.0 seconds

$ make check AM_LAZY_CHECK=yes                  # 1 time
9 seconds

These numbers are still acceptable IMHO, and are a price worth to paying
for the improved cleanliness of the code.

Then again, if I modify the assignments of the am__test_logs, am__test_bases
and am__test_results variables to be immediate-evaluation assignments (i.e.,
defined with ":=" rather than "="), the performance becomes on-par with that
before the series -- in some cases, actually *better*:

====================================================================
 AFTER the series has been applied, with immediate-evaluation tweak
====================================================================

$ make all                                      # 12 times
2.5 seconds

$ make recheck                                  # 1 time
1.8 seconds

$ make recheck TESTS='1 2 3 4 5 6 7 8 9'        # 6 times
2.0 seconds

$ make check AM_LAZY_CHECK=yes                  # 1 time
4 seconds

The downside of immediate evaluation is that, to ensure immediate-eval
variables are faithful w.r.t. modifications of the lazy-eval ones they
reference, any definition of an immediate-eval variable should must
appear *after* all the definitions of the lazy-eval variables.  And in
fact, to make the tweak referenced above actually work, I had to manually
move the definition of TESTS *before* the definitions of am__test_logs,
am__test_bases and am__test_results.

This further Automake-implemented variable reordering will require some
support in the Automake machinery that handles variable definitions.
We'll have to ensure that any definition of immediate-eval variables
coming from Automake (either the script or the Automake-provided '.am'
fragments) gets placed *after* all the lazy-eval variables defined by
Automake, as well as after any variable defined by the user (either
lazy or immediate).  That is something I'd rather do after my current
work on the 'experimental/ng/var-simplify' branch is completed though.

So, in conclusion: while I think we should keep an eye on performance,
I believe this series should be merged as-is, and optimizations should
be done later (once we'll have more framework about variable handling
in place).

WDYT?

Regards,
  Stefano