Re: Documentation for the parallel-tests driver. [4/4]

[Ralf Wildenhues]

* Akim Demaille wrote on Thu, Oct 16, 2008 at 05:16:20PM CEST:
> >>> "RW" == Ralf Wildenhues <address@hidden> writes:
> 
>  > address@hidden Simple tests using @samp{parallel-tests}
>  > address@hidden @option{parallel-tests}, Using
>  > +The option @option{parallel-tests} (@pxref{Options}) enables a test
>  > +suite driver that is mostly compatible to the simple test driver
>  > +described above, but provides a few more features and slightly different
>  > +semantics.  It features concurrent execution of tests with @code{make -j},
>  > +allows to specify inter-test dependencies, lazy reruns of tests that
>  > +have not completed in a prior run,
> 
> That's one use, but more importantly, this is perfect for unit-tests.
> Unit-tests are often self-contained, in which case if none of their
> dependencies changed, there is no point in running the test again, the
> results will be the same.

Yes.  The first paragraph is intended as a feature overview only.  Unit
tests are mentioned later:

address@hidden Unit tests
+The combination of lazy test execution and correct dependencies between
+tests and their sources may be exploited for efficient unit testing
+during development.  To further speed up the edit-compile-test cycle, it
+may even be useful to specify compiled programs in @code{EXTRA_PROGRAMS}
+instead of with @code{check_PROGRAMS}, as the former allows intertwined
+compilation and test execution (but note that @code{EXTRA_PROGRAMS} are
+not cleaned automatically, @pxref{Uniform}).

I think this is special enough that it doesn't need to be mentioned
early.

>  > address@hidden VERBOSE
>  > +As with the simple driver above, by default one status line is printed
>  > +per completed test, and a short summary after the suite has completed.
>  > +If the variable @samp{VERBOSE} is set, the @file{test-suite.log} file is
>  > +appended after the summary.
> 
> IMHO, you should emphasize that tests should be verbose by default.
> VERBOSE is no longer an envvar that test should read to decide whether
> displaying something or not: they should be verbose by default, it
> goes into the log file.  Now VERBOSE is meant for "make check" itself.

Good point; thanks.

> I think that you should detail how to write a .test->.log rule,
> showing the default value for instance (I use GNU Make syntax).
> 
>   # From a test file to a log file.
>   # Do not use a regular `.test.log:' rule here, since in that case the
>   # following rule (without incoming extension) will mask this one.
>   %.log: %.test $(check_programs)
>       @$(am__check_pre) $${dir}$< $(am__check_post)
> 
> because that's where the user can explain how the test is run.

Erm, the framework in Automake is _not_ supposed to allow the user to
write such rules herself.  That would likely be problematic, the least
of which being that it's difficult to make Automake version-agnostic.

Consequently, there is little point in showing users how to write such
a rule.

> Something I dislike very much in the current framework for tests in
> Automake is that each test must be an executable, so it is very
> inconvenient to test a given program foo with many different inputs.
> In that case I use
> 
>    TESTS = 1.foo 2.foo 3.foo ...
> 
> and
> 
>   %.log: %.test $(check_programs)
>       @$(am__check_pre) $(top_builddir)/src/foo $${dir}$< $(am__check_post)
> 
> i.e., the testee is foo, and TESTS are plain files, not a bazillion of
> 1.sh that simple "exec $(top_builddir)/src/foo 1.foo".

Try
  TESTS_ENVIRONMENT = $(top_builddir)/src/foo

and adding $(top_builddir)/src/foo to check_SCRIPTS.

If you have only some files that are to be run with foo, and some not,
consider looking at coreutils generalization of this:
<http://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=blob;f=tests/check.mk;hb=HEAD>.

Cheers,
Ralf