bug#11287: Various issues with the test suite framework

[Stefano Lattarini]

On 04/20/2012 01:13 PM, Akim Demaille wrote:
> Hi!
>
Hi Akim, sorry for the delay.

> I have seen that check-html will be removed, or rather moved
> into the contrib part, but there are a few issues:
>
> - the target is not declared recursive, so one has to write
>  the bouncing target herself.
> 
> - because of that, the "naive" implementation of check-html
>  that just bounces the right directories will miss the
>  dependencies that check features (in other words, make
>  check properly recursively builds before, while check-html
>  would just go into the typical tests/ directory, and then
>  bounce to "check", so it completely missed the recursion
>  in the other directories and their dependencies).
>
Wasn't this issue already present in Automake 1.11.1?  (So, not a
regression, unless I'm mistaken).  In any case, a test case
exposing the issue would be appreciated, if you have one at hand.

> - the .log.html recipe features $$RST2HTML, but in '', so it is
>  useless.  And anyway $(RST2HTML) already does the job.
>
> - "$(AM_RST2HTMLFLAGS) $(RST2HTMLFLAGS)" is missing.
>
None of these are covered by the Automake test suite sadly, and I
see them as really low low priority anyway, so I'm not going to
tackle them right now.  But I'll gladly accept patches by anyone
who has more knowledge or interest than me in this area ;-)

> - RECURSIVE_TARGETS cannot be highjacked by the users, that's
>  sad.  Could a $(USER_RECURSIVE_TARGETS) be added to
>  RECURSIVE_TARGETS so that we can add our own targets that
>  traverse the whole package?
>
There had been an attempt to offer an interface for user-defined
recursive rules already:

  <http://thread.gmane.org/gmane.comp.sysutils.automake.general/11890>

But unfortunately the discussion stagnated and died out, and I don't
have time to resurrect it nor the relevant patches ATM.  Still, it
would be a cool feature to have, so any help would be appreciated.

>  Actually maybe this should
>  be an Automake option to add recursive targets, and
>  then TARGET-am and TARGET-recursive would be generated and
>  bound to the proper TARGET.
> 

> - the test compiler does not appear as a dependency for
>   the test logs that use it.
>
This makes sense, since the test compiler is not required to be an
in-project script, and not even to be specified as an absolute path
-- it can be a command like "sh" or "/usr/bin/env perl".

> - it would be equally useful to be able to add dependencies
>   to a category of test drivers.  Say, with the documented
>   example:
> 
>     TESTS = foo.pl bar.py baz
>     TEST_EXTENSIONS = .pl .py
>     PL_LOG_COMPILER = $(PERL)
>     AM_PL_LOG_FLAGS = -w
>     PY_LOG_COMPILER = $(PYTHON)
>     AM_PY_LOG_FLAGS = -v
>     LOG_COMPILER = ./wrapper-script
>     AM_LOG_FLAGS = -d
> 
>  be able to add PL_LOG_DEPENDENCIES, PY_LOG_DEPENDENCIESS and
>  LOG_DEPENDENCIES.
>
This seems like a cool feature, but not trivial to implement in the current
Automake code base.  Still, I have some half-finished patches to rework
the parallel-tests harness in Automake-NG (reviews welcome, BTW ;-), and
once those are in place, implementing your proposed feature in Automake-NG
might be very easy.

> I don't think EXTRA dependencies would be useful.
>
Right.  If we'll ever need automatically computed dependencies, we should
name them AM_PL_LOG_DEPENDENCIES etc., without impinging on the user
namespace.

Thanks,
  Stefano