[PATCH v4 2/3] parallel-tests: allow custom driver scripts

[Stefano Lattarini]

Allow suffix-based definition of custom "driver script" for the
test scripts.  These driver scripts will be responsible of
launching the tests (or their corresponding $(LOG_COMPILER), if
they have an associated one), interpreting and displaying the
test results, and writing the `.log' files.

This new API should allow easy and flexible use of different test
protocols in the future; in particular, we plan to use it will be
used to implement TAP and SubUnit harnesses.

* automake.in (handle_tests): Define default for $(LOG_DRIVER),
and, for any registered text extension `<ext>', define defaults
for $(<ext>_LOG_DRIVER).  Substitute %DRIVER% using these new
variables, instead of the old internal $(am__pt_wrap).  When
processing check2.am, also substitute %DRIVER_FLAGS%.
Require auxiliary script `pt-driver' only if no wrapper has been
explicitly defined for the test script kinds.
* am/check2.am (?GENERIC?%EXT%$(EXEEXT).log, ?GENERIC?%EXT%.log,
?!GENERIC?%OBJ%): Pass the %DRIVER_FLAGS% to the %DRIVER% call.
* doc/automake.texi (Custom Test Drivers): New section and node.
(Overview of Custom Test Drivers Support): New subsection.
(Declaring Custom Test Drivers in @file{Makefile.am}): Likewise.
(APIs for Custom Test Drivers): New subsection, still incomplete,
with sketchy descriptions only written in texinfo comments.
(Auxiliary Programs): Mention the new `pt-driver' script.
(Optional): Mention `pt-driver' in AC_CONFIG_AUX_DIR.  Since we
are at it, break the list of auxiliary script by placing one per
line, to simplify potential future additions of new scripts.
* tests/parallel-tests-no-pt-driver.test: New test.
* tests/test-driver-custom.test: Likewise.
* tests/test-driver-custom-xfail-tests.test: Likewise.
* tests/test-driver-fail.test: Likewise.
* tests/Makefile.am: Update.
* NEWS: Update.
---
 ChangeLog                                  |   36 ++++++
 NEWS                                       |   12 ++-
 automake.in                                |   38 +++++-
 doc/automake.texi                          |  176 +++++++++++++++++++++++++++-
 lib/Automake/tests/Makefile.in             |    4 +-
 lib/am/check2.am                           |    4 +-
 tests/Makefile.am                          |    4 +
 tests/Makefile.in                          |    8 +-
 tests/test-driver-custom-no-pt-driver.test |   63 ++++++++++
 tests/test-driver-custom-xfail-tests.test  |  147 +++++++++++++++++++++++
 tests/test-driver-custom.test              |  140 ++++++++++++++++++++++
 tests/test-driver-fail.test                |   65 ++++++++++
 12 files changed, 675 insertions(+), 22 deletions(-)
 create mode 100755 tests/test-driver-custom-no-pt-driver.test
 create mode 100755 tests/test-driver-custom-xfail-tests.test
 create mode 100755 tests/test-driver-custom.test
 create mode 100755 tests/test-driver-fail.test

diff --git a/ChangeLog b/ChangeLog
index 0771b24..d84355a 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,5 +1,41 @@
 2011-06-15  Stefano Lattarini  <address@hidden>
 
+       parallel-tests: allow custom driver scripts
+       Allow suffix-based definition of custom "driver script" for the
+       test scripts.  These driver scripts will be responsible of
+       launching the tests (or their corresponding $(LOG_COMPILER), if
+       they have an associated one), interpreting and displaying the
+       test results, and writing the `.log' files.
+       This new API should allow easy and flexible use of different test
+       protocols in the future; in particular, we plan to use it will be
+       used to implement TAP and SubUnit harnesses.
+       * automake.in (handle_tests): Define default for $(LOG_DRIVER),
+       and, for any registered text extension `<ext>', define defaults
+       for $(<ext>_LOG_DRIVER).  Substitute %DRIVER% using these new
+       variables, instead of the old internal $(am__pt_wrap).  When
+       processing check2.am, also substitute %DRIVER_FLAGS%.
+       Require auxiliary script `pt-driver' only if no wrapper has been
+       explicitly defined for the test script kinds.
+       * am/check2.am (?GENERIC?%EXT%$(EXEEXT).log, ?GENERIC?%EXT%.log,
+       ?!GENERIC?%OBJ%): Pass the %DRIVER_FLAGS% to the %DRIVER% call.
+       * doc/automake.texi (Custom Test Drivers): New section and node.
+       (Overview of Custom Test Drivers Support): New subsection.
+       (Declaring Custom Test Drivers in @file{Makefile.am}): Likewise.
+       (APIs for Custom Test Drivers): New subsection, still incomplete,
+       with sketchy descriptions only written in texinfo comments.
+       (Auxiliary Programs): Mention the new `pt-driver' script.
+       (Optional): Mention `pt-driver' in AC_CONFIG_AUX_DIR.  Since we
+       are at it, break the list of auxiliary script by placing one per
+       line, to simplify potential future additions of new scripts.
+       * tests/parallel-tests-no-pt-driver.test: New test.
+       * tests/test-driver-custom.test: Likewise.
+       * tests/test-driver-custom-xfail-tests.test: Likewise.
+       * tests/test-driver-fail.test: Likewise.
+       * tests/Makefile.am: Update.
+       * NEWS: Update.
+
+2011-06-15  Stefano Lattarini  <address@hidden>
+
        parallel-tests: add auxiliary script 'pt-driver', refactor
        This refactoring should cause no API of functionality change,
        and is meant only to simplify the future implementation of TAP
diff --git a/NEWS b/NEWS
index 6db2fe0..2be5a81 100644
--- a/NEWS
+++ b/NEWS
@@ -13,9 +13,15 @@ New in 1.11a:
 
 * Changes to Automake-generated testsuite harnesses:
 
-  - The parallel-tests driver is now implemented (partly at least) with
-    the help of automake-provided auxiliary scripts (e.g., `pt-driver'),
-    instead of relying entirely on code in the generated Makefile.in.
+  - The default testsuite driver offered by the 'parallel-tests' option is
+    now implemented (partly at least) with the help of automake-provided
+    auxiliary scripts (e.g., `pt-driver'), instead of relying entirely on
+    code in the generated Makefile.in.
+
+  - The package authors can now use customary testsuite drivers within
+    the framework provided by the 'parallel-tests' testsuite harness.
+    Consistently with the existing syntax, this can be done by defining
+    special makefile variables `LOG_DRIVER' and `<ext>_LOG_DRIVER'.
 
 * Miscellaneous changes:
 
diff --git a/automake.in b/automake.in
index 7eb5464..153d3d5 100644
--- a/automake.in
+++ b/automake.in
@@ -4991,10 +4991,6 @@ sub handle_tests
 
       if (my $parallel_tests = option 'parallel-tests')
         {
-          require_conf_file ($parallel_tests->{position}, FOREIGN, 
'pt-driver');
-          define_variable ('am__pt_driver',
-                           "\$(SHELL) $am_config_aux_dir/pt-driver",
-                           INTERNAL);
          define_variable ('TEST_SUITE_LOG', 'test-suite.log', INTERNAL);
          define_variable ('TEST_SUITE_HTML', '$(TEST_SUITE_LOG:.log=.html)', 
INTERNAL);
          my $suff = '.test';
@@ -5038,6 +5034,19 @@ sub handle_tests
                      if substr ($obj, - length ($test_suffix)) eq $test_suffix;
                  }
                $obj .= '.log';
+                # The "test driver" program, deputed to handle tests protocol 
used by
+                # test scripts.  By default, it's assumed that no protocol is 
used,
+                # so we fall back to the old "parallel-tests" behaviour, 
implemented
+                # by the `pt-driver' auxiliary script.
+                if (! var 'LOG_DRIVER')
+                  {
+                    require_conf_file ($parallel_tests->{position}, FOREIGN, 
'pt-driver');
+                    define_variable ('LOG_DRIVER',
+                                     "\$(SHELL) $am_config_aux_dir/pt-driver",
+                                     INTERNAL);
+                  }
+               my $driver = '$(LOG_DRIVER)';
+               my $driver_flags = '$(AM_LOG_DRIVER_FLAGS) $(LOG_DRIVER_FLAGS)';
                my $compile = 'LOG_COMPILE';
                define_variable ($compile,
                                 '$(LOG_COMPILER) $(AM_LOG_FLAGS) 
$(LOG_FLAGS)', INTERNAL);
@@ -5045,7 +5054,8 @@ sub handle_tests
                                                GENERIC => 0,
                                                OBJ => $obj,
                                                SOURCE => $val,
-                                               DRIVER => '$(am__pt_driver)',
+                                               DRIVER => $driver,
+                                               DRIVER_FLAGS => $driver_flags,
                                                COMPILE =>'$(' . $compile . ')',
                                                EXT => '',
                                                am__EXEEXT => 'FALSE');
@@ -5076,6 +5086,17 @@ sub handle_tests
                {
                  (my $ext = $test_suffix) =~ s/^\.//;
                  $ext = uc $ext;
+                  # See comments about definition of LOG_DRIVER, above.
+                  if (! var "${ext}_LOG_DRIVER")
+                  {
+                    require_conf_file ($parallel_tests->{position}, FOREIGN, 
'pt-driver');
+                    define_variable ("${ext}_LOG_DRIVER",
+                                     "\$(SHELL) $am_config_aux_dir/pt-driver",
+                                     INTERNAL);
+                  }
+                 my $driver = '$(' . $ext . '_LOG_DRIVER)';
+                 my $driver_flags = '$(AM_' . $ext . '_LOG_DRIVER_FLAGS) ' .
+                                     '$(' . $ext . '_LOG_DRIVER_FLAGS)';
                  my $compile = $ext . '_LOG_COMPILE';
                  define_variable ($compile,
                                   '$(' . $ext . '_LOG_COMPILER) $(AM_' .  $ext 
. '_LOG_FLAGS)'
@@ -5085,16 +5106,19 @@ sub handle_tests
                                                  GENERIC => 1,
                                                  OBJ => '',
                                                  SOURCE => '$<',
-                                                 DRIVER => '$(am__pt_driver)',
+                                                 DRIVER => $driver,
+                                                 DRIVER_FLAGS => $driver_flags,
                                                  COMPILE => '$(' . $compile . 
')',
                                                  EXT => $test_suffix,
                                                  am__EXEEXT => $am_exeext);
                }
            }
 
+         # FIXME: this is partly out-of-date w.r.t. the rest of the
+          # FIXME: code now ... what is the best fix?
          define_variable ('TEST_LOGS_TMP', '$(TEST_LOGS:.log=.log-t)', 
INTERNAL);
-
          $clean_files{'$(TEST_LOGS_TMP)'} = MOSTLY_CLEAN;
+
          $clean_files{'$(TEST_LOGS)'} = MOSTLY_CLEAN;
          $clean_files{'$(TEST_SUITE_LOG)'} = MOSTLY_CLEAN;
          $clean_files{'$(TEST_SUITE_HTML)'} = MOSTLY_CLEAN;
diff --git a/doc/automake.texi b/doc/automake.texi
index 032e9ec..7bbdd57 100644
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -317,6 +317,7 @@ Support for test suites
 
 * Simple Tests::                Listing programs and scripts in @code{TESTS}
 * Simple Tests using parallel-tests::  More powerful test driver
+* Custom Test Drivers::         Writing and using custom test drivers
 * DejaGnu Tests::               Interfacing with the external testing framework
 * Install Tests::               Running tests on installed packages
 
@@ -2294,6 +2295,10 @@ longer installed automatically, and it should be safe to 
remove it.
 @item py-compile
 This is used to byte-compile Python scripts.
 
address@hidden compile
+This implements the default testsuite driver offered by the
address@hidden testsuite harness.
+
 @item symlink-tree
 This program duplicates a tree of directories, using symbolic links
 instead of copying files.  Such an operation is performed when building
@@ -2906,12 +2911,24 @@ The Autoconf Manual}.
 @item AC_CONFIG_AUX_DIR
 Automake will look for various helper scripts, such as
 @file{install-sh}, in the directory named in this macro invocation.
address@hidden This list is accurate relative to version 1.8
-(The full list of scripts is: @file{ar-lib}, @file{config.guess},
address@hidden, @file{depcomp}, @file{elisp-comp}, @file{compile},
address@hidden, @file{ltmain.sh}, @file{mdate-sh}, @file{missing},
address@hidden, @file{py-compile}, @file{texinfo.tex}, and
address@hidden)  Not all scripts are always searched for; some scripts
address@hidden This list is accurate relative to version 1.11
+(The full list of scripts is:
address@hidden,
address@hidden,
address@hidden,
address@hidden,
address@hidden,
address@hidden,
address@hidden,
address@hidden,
address@hidden,
address@hidden,
address@hidden,
address@hidden,
address@hidden,
address@hidden,
address@hidden)
+Not all scripts are always searched for; some scripts
 will only be sought if the generated @file{Makefile.in} requires them.
 
 If @code{AC_CONFIG_AUX_DIR} is not given, the scripts are looked for in
@@ -8600,6 +8617,7 @@ are very similar.
 @menu
 * Simple Tests::                Listing programs and scripts in @code{TESTS}
 * Simple Tests using parallel-tests::  More powerful test driver
+* Custom Test Drivers::         Writing and using custom test drivers
 * DejaGnu Tests::               Interfacing with the external testing framework
 * Install Tests::               Running tests on installed packages
 @end menu
@@ -8926,6 +8944,152 @@ semantics of FreeBSD and OpenBSD @command{make} 
conflict with this).
 In case of doubt you may want to require to use GNU @command{make},
 or work around the issue with inference rules to generate the tests.
 
address@hidden Custom Test Drivers
address@hidden Custom Test Drivers
+
address@hidden
+* Overview of Custom Test Drivers Support::
+* Declaring Custom Test Drivers::
+* API for Custom Test Drivers::
address@hidden menu
+
address@hidden Overview of Custom Test Drivers Support
address@hidden Overview of Custom Test Drivers Support
+
+Starting form version 1.12, the @option{parallel-tests} harness has
+been augmented to allow the package authors to use their own test
+drivers, in case the
address@hidden FIXME: this should become "default ones" once we have TAP and 
Subunit
+default one
+is inadequate for their purposes.
address@hidden FIXME: add this once we have TAP and Subunit
address@hidden "or do not support their testing protocol of choice."
+
+A custom test driver is expected to properly run the test programs passed
+to it (bearing in mind that these programs are allowed to be complemented
+by command-line arguments), to analyze their execution and outcome, to
+create the log files associated to these test runs, and to display the
+test results on the console.  It is the sole responsibility of the author
+of the test driver to ensure that it implements all the above steps
+meaningfully and correctly.  Automake is not nor can be of any help here.
+
+The details of how the analysis of a test script's result is to be
+conducted is left to the individual drivers.  Some drivers might only
+consider the test script exit status (this is done for example by the
+default test driver used by the @option{parallel-tests} harness).
+Other drivers might implement more complex and advanced test protocols,
+which might require them to parse and interpreter the output emitted by
+a test script (examples of such protocols are TAP and SubUnit).
address@hidden FIXME: add links?
+
+It's very important to note that, even when using custom test drivers,
+most of the @option{parallel-tests} infrastructure described in the
+previous section remains deliberately in place; this comprises concurrency
+through the use of  @command{make}'s @option{-j} option, inter-test
+dependencies, lazy reruns of tests, @code{check_*} variables, summary and
+verbose output in @samp{RST} (reStructuredText) format, and the definition
+and honoring of @code{TESTS_ENVIRONMENT} and @code{AM_TESTS_ENVIRONMENT}
+variables, and of generic and extension-specific @code{LOG_COMPILER}
+and @code{LOG_FLAGS} variables,  On the other hand, the exact semantics of
+how (and if) @option{color-tests}, @code{XFAIL_TESTS}, and hard errors
+are supported and handled is left to the individual test drivers.
+
address@hidden TODO: We should really add a working example in the doc/ 
directory,
address@hidden TODO: and reference if from here.
+
address@hidden Declaring Custom Test Drivers
address@hidden Declaring Custom Test Drivers in @file{Makefile.am}
+
address@hidden _LOG_DRIVER
address@hidden _LOG_DRIVER_FLAGS
address@hidden LOG_DRIVER
address@hidden LOG_DRIVER_FLAGS
address@hidden @var{ext}_LOG_DRIVER
address@hidden @var{ext}_LOG_DRIVER_FLAGS
address@hidden address@hidden
address@hidden AM_LOG_DRIVER_FLAGS
+Custom testsuite drivers are declared by defining the make variables
address@hidden or @address@hidden (where @var{ext} must
+be declared in @code{TEST_EXTENSIONS}).  They will be used to drive the
+execution, logging, and outcome report of the tests with corresponding
+extensions, or of those with no registered extension in the case of
address@hidden  Clearly, multiple distinct test drivers can be
+declared in the same @file{Makefile.am}.  Note moreover that the
address@hidden variables are @emph{not} a substitute for the
address@hidden variables: the two sets of variables can, and
+often do, usefully and legitimately coexist.
+
address@hidden TODO: We should really be able to point to a clarifying example 
here!
+
+The developer-reserved variable @code{AM_LOG_DRIVER_FLAGS} and the
+user-reserved variable @code{LOG_DRIVER_FLAGS} can be used to define
+flags that will be passed to each invocation of @code{LOG_DRIVER}, with
+the use-defined flags taking precedence over the developer-reserved ones.
+
+Similarly, for each @var{ext} declared in @code{TEST_EXTENSIONS},
+flags listed in @address@hidden and
address@hidden@var{ext}_LOG_DRIVER_FLAGS} will be passed to invocations
+of @address@hidden
+
address@hidden API for Custom Test Drivers
address@hidden API for Custom Test Drivers
+
+Here we'll describe the APIs currently offered to the authors of custom
+test drivers.  Note that they are still somewhat experimental, and might
+undergo changes and tightenings in the future, to accommodate for new
+features or to satisfy additional portability requirements.
+
+Also, note that the main characteristic of these APIs is that they are
+designed to share as much infrastructure, semantics, and implementation
+details as possible with the plain @option{parallel-tests} harness.  So
+everything said in the previous section should apply here too, unless
+explicitly noted otherwise.
+
+A custom driver can relay on the following options and arguments being
+passed to it automatically by the Automake's @option{parallel-tests}
+harness.  It is @emph{mandatory} that it understands all of them (even
+if the exact interpretation of the associated semantics can change
+between a test driver and another, and even be a no-op in some drivers).
+
address@hidden  @i{Options and flags that the driver must handle.  Generation
+of ``.log'' files.  Console output the driver is expected to produce.
+Support for colored output, XFAIL_TESTS, and DISABLE_HARD_ERRORS}
+
address@hidden
address@hidden The driver script should follow a simple protocol in order to 
really
address@hidden work with Automake-generated test harnesses:
address@hidden
address@hidden  * options that *must* be supported by the driver:
address@hidden     --test-name=NAME
address@hidden     --log-file=PATH
address@hidden     --color-tests={yes|no}
address@hidden     --enable-hard-errors={yes|no}
address@hidden     --expect-failure={yes|no}
address@hidden    The path specified by the `--log-file' *must* be correctly 
created
address@hidden    by the driver; the format of what gets written in there, as 
well as
address@hidden    the exact semantic of the other options, might be variable.  
In
address@hidden    particular, the driver is free to consider some of those 
options as
address@hidden    no-ops (but must recognize them nonetheless).
address@hidden
address@hidden  * Generated files: only the proper `.log' should be created 
(but the
address@hidden    driver can use temporary files if it needs to, only it should 
clean
address@hidden    them up properly).
address@hidden
address@hidden  * driver-specific options (AM_LOG_DRIVER_FLAGS and 
LOG_DRIVER_FLAGS)
address@hidden    that get passed to the driver script by the Makefile.
address@hidden
address@hidden  * what to exactly do with xfailing tests and disable-hard-errors
address@hidden    directive is up to the driver script, and might depend from 
the
address@hidden    test protocol it implements.
address@hidden
address@hidden  * explain what kind of console output the driver is expected to 
produce;
address@hidden    it can have some leeway here, but IMHO should provide a "look 
& feel"
address@hidden    similar to the existing "parallel-tests" output.
address@hidden
address@hidden  * not strictly necessary for the driver to to honour 
color-tests or
address@hidden    DISABLE_HARD_ERRORS; but usually it's nice to do so, in oder 
to provide
address@hidden    a better user experience.
+
 
 @node DejaGnu Tests
 @section DejaGnu Tests
diff --git a/lib/Automake/tests/Makefile.in b/lib/Automake/tests/Makefile.in
index d3ed3dd..5bc86bf 100644
--- a/lib/Automake/tests/Makefile.in
+++ b/lib/Automake/tests/Makefile.in
@@ -165,11 +165,11 @@ esac;                                                     
\
 $(AM_TESTS_ENVIRONMENT) $(TESTS_ENVIRONMENT)
 RECHECK_LOGS = $(TEST_LOGS)
 AM_RECURSIVE_TARGETS = check check-html recheck recheck-html
-am__pt_driver = $(SHELL) $(top_srcdir)/lib/pt-driver
 TEST_SUITE_LOG = test-suite.log
 TEST_SUITE_HTML = $(TEST_SUITE_LOG:.log=.html)
 am__test_logs1 = $(TESTS:=.log)
 TEST_LOGS = $(am__test_logs1:.pl.log=.log)
+PL_LOG_DRIVER = $(SHELL) $(top_srcdir)/lib/pt-driver
 PL_LOG_COMPILE = $(PL_LOG_COMPILER) $(AM_PL_LOG_FLAGS) $(PL_LOG_FLAGS)
 TEST_LOGS_TMP = $(TEST_LOGS:.log=.log-t)
 DISTFILES = $(DIST_COMMON) $(DIST_SOURCES) $(TEXINFOS) $(EXTRA_DIST)
@@ -462,7 +462,7 @@ recheck recheck-html:
        $(MAKE) $(AM_MAKEFLAGS) $$target AM_MAKEFLAGS='$(AM_MAKEFLAGS) 
TEST_LOGS="'"$$list"'"'
 .pl.log:
        @p='$<'; $(am__check_pre) \
-       $(am__pt_driver) $(am__test_driver_flags) -- \
+       $(PL_LOG_DRIVER) $(am__test_driver_flags) $(AM_PL_LOG_DRIVER_FLAGS) 
$(PL_LOG_DRIVER_FLAGS) -- \
        $(PL_LOG_COMPILE) "$$tst"
 
 distdir: $(DISTFILES)
diff --git a/lib/am/check2.am b/lib/am/check2.am
index f3116c8..b7ed0ee 100644
--- a/lib/am/check2.am
+++ b/lib/am/check2.am
@@ -18,7 +18,7 @@
 ?GENERIC?%EXT%.log:
 ?!GENERIC?%OBJ%: %SOURCE%
        @p='%SOURCE%'; $(am__check_pre) \
-       %DRIVER% $(am__test_driver_flags) -- \
+       %DRIVER% $(am__test_driver_flags) %DRIVER_FLAGS% -- \
        %COMPILE% "$$tst"
 
 ## If no programs are built in this package, then this rule is removed
@@ -28,6 +28,6 @@
 if %am__EXEEXT%
 ?GENERIC?%EXT%$(EXEEXT).log:
        @p='%SOURCE%'; $(am__check_pre) \
-       %DRIVER% $(am__test_driver_flags) -- \
+       %DRIVER% $(am__test_driver_flags) %DRIVER_FLAGS% -- \
        %COMPILE% "$$tst"
 endif %am__EXEEXT%
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 8efefab..36445b5 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -719,6 +719,10 @@ parallel-tests-subdir.test \
 parallel-tests-interrupt.test \
 parallel-tests-reset-term.test \
 parallel-tests-pt-driver.test \
+test-driver-custom-no-pt-driver.test \
+test-driver-custom.test \
+test-driver-custom-xfail-tests.test \
+test-driver-fail.test \
 parse.test \
 percent.test \
 percent2.test \
diff --git a/tests/Makefile.in b/tests/Makefile.in
index 14fcf7b..df2a818 100644
--- a/tests/Makefile.in
+++ b/tests/Makefile.in
@@ -170,12 +170,12 @@ esac;                                                     
\
 $(AM_TESTS_ENVIRONMENT) $(TESTS_ENVIRONMENT)
 RECHECK_LOGS = $(TEST_LOGS)
 AM_RECURSIVE_TARGETS = check check-html recheck recheck-html
-am__pt_driver = $(SHELL) $(top_srcdir)/lib/pt-driver
 TEST_SUITE_LOG = test-suite.log
 TEST_SUITE_HTML = $(TEST_SUITE_LOG:.log=.html)
 TEST_EXTENSIONS = .test
 am__test_logs1 = $(TESTS:=.log)
 TEST_LOGS = $(am__test_logs1:.test.log=.log)
+TEST_LOG_DRIVER = $(SHELL) $(top_srcdir)/lib/pt-driver
 TEST_LOG_COMPILE = $(TEST_LOG_COMPILER) $(AM_TEST_LOG_FLAGS) \
        $(TEST_LOG_FLAGS)
 TEST_LOGS_TMP = $(TEST_LOGS:.log=.log-t)
@@ -973,6 +973,10 @@ parallel-tests-subdir.test \
 parallel-tests-interrupt.test \
 parallel-tests-reset-term.test \
 parallel-tests-pt-driver.test \
+test-driver-custom-no-pt-driver.test \
+test-driver-custom.test \
+test-driver-custom-xfail-tests.test \
+test-driver-fail.test \
 parse.test \
 percent.test \
 percent2.test \
@@ -1481,7 +1485,7 @@ recheck recheck-html:
        $(MAKE) $(AM_MAKEFLAGS) $$target AM_MAKEFLAGS='$(AM_MAKEFLAGS) 
TEST_LOGS="'"$$list"'"'
 .test.log:
        @p='$<'; $(am__check_pre) \
-       $(am__pt_driver) $(am__test_driver_flags) -- \
+       $(TEST_LOG_DRIVER) $(am__test_driver_flags) $(AM_TEST_LOG_DRIVER_FLAGS) 
$(TEST_LOG_DRIVER_FLAGS) -- \
        $(TEST_LOG_COMPILE) "$$tst"
 
 distdir: $(DISTFILES)
diff --git a/tests/test-driver-custom-no-pt-driver.test 
b/tests/test-driver-custom-no-pt-driver.test
new file mode 100755
index 0000000..f4c996e
--- /dev/null
+++ b/tests/test-driver-custom-no-pt-driver.test
@@ -0,0 +1,63 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Check that auxiliary script 'pt-driver' doesn't get needlessly
+# installed or referenced if it's not used, i.e., if the user has
+# defined his own `*LOG_DRIVER' variables.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in <<'END'
+AC_PROG_CC
+AC_CONFIG_FILES([sub1/Makefile sub2/Makefile])
+AC_SUBST([X_LOG_DRIVER], [none])
+AC_OUTPUT
+END
+
+mkdir sub1 sub2
+
+cat > Makefile.am <<'END'
+SUBDIRS = sub1 sub2
+LOG_DRIVER =
+TEST_LOG_DRIVER = :
+TESTS = foo bar.test
+END
+
+cat > sub1/Makefile.am <<'END'
+TEST_EXTENSIONS = .x .sh .pl
+SH_LOG_DRIVER = dummy1
+PL_LOG_DRIVER = dummy2
+TESTS = a.pl b.sh c.x
+END
+
+cat > sub2/Makefile.am <<'END'
+TEST_EXTENSIONS = .bar
+LOG_DRIVER = x
+BAR_LOG_DRIVER = y
+TESTS = 1 2.bar 3.test 4.t 5.tt $(check_PROGRAMS)
+check_PROGRAMS = p1 p2$(EXEEXT) p3.bar p4.suf
+END
+
+$ACLOCAL
+
+for opts in '' '--add-missing' '-a -c'; do
+  $AUTOMAKE $opts
+  $FGREP pt-driver Makefile.in sub1/Makefile.in sub2/Makefile.in && Exit 1
+  find . | $FGREP pt-driver && Exit 1
+done
+
+:
diff --git a/tests/test-driver-custom-xfail-tests.test 
b/tests/test-driver-custom-xfail-tests.test
new file mode 100755
index 0000000..0d10594
--- /dev/null
+++ b/tests/test-driver-custom-xfail-tests.test
@@ -0,0 +1,147 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Custom test drivers: "abstract" XFAIL_TESTS support.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in <<'END'
+AC_SUBST([nihil], [])
+AC_SUBST([ac_xfail_tests], ['x5.test x6$(test_suffix)'])
+AC_CONFIG_FILES([sub1/Makefile sub2/Makefile])
+AC_OUTPUT
+END
+
+cat > Makefile.am <<'END'
+SUBDIRS = . sub1 sub2
+TEST_LOG_DRIVER = $(srcdir)/td
+TESTS = pass.test xfail.test
+XFAIL_TESTS = xfail.test
+END
+
+mkdir sub1 sub2
+
+cat > sub1/Makefile.am <<END
+empty =
+
+TEST_LOG_DRIVER = \$(top_srcdir)/td
+
+# XFAIL_TESTS should gracefully handle TAB characters, and multiple
+# whitespaces.
+XFAIL_TESTS =\$(empty)${tab}x1.test x2.test${tab}x3.test${tab}\
+x4.test ${tab} x5.test              x6.test${tab}\$(empty)
+
+TESTS = pass.test x1.test x2.test x3.test x4.test x5.test x6.test
+END
+
+cat > sub2/Makefile.am <<'END'
+AUTOMAKE_OPTIONS = -Wno-portability-recursive
+
+TEST_LOG_DRIVER = $(srcdir)/../td
+
+# XFAIL_TESTS should gracefully AC_SUBST @substitution@ and
+# make variables indirections.
+an_xfail_test = x1.test
+test_suffix = .test
+v0 = x3.test
+v1 = v
+v2 = 0
+XFAIL_TESTS = $(an_xfail_test) x2.test @nihil@ x3${test_suffix}
+XFAIL_TESTS += $($(v1)$(v2)) x4.test @ac_xfail_tests@
+
+TESTS = pass.test x1.test x2.test x3.test x4.test x5.test x6.test
+END
+
+cat > pass.test <<'END'
+#!/bin/sh
+exit 0
+END
+
+cat > xfail.test <<'END'
+#!/bin/sh
+exit 1
+END
+
+chmod a+x pass.test xfail.test
+
+cp pass.test sub1/pass.test
+cp pass.test sub2/pass.test
+
+for i in 1 2 3 4 5 6; do
+  cp xfail.test sub1/x$i.test
+  cp xfail.test sub2/x$i.test
+done
+
+cat > td <<'END'
+#! /bin/sh
+set -e
+test_name=INVALID
+log_file=/dev/null
+expect_failure=no
+while test $# -gt 0; do
+  case $1 in
+    --test-name) test_name=$2; shift;;
+    --expect-failure) expect_failure=$2; shift;;
+    --log-file) log_file=$2; shift;;
+    # Ignored.
+    --color-tests) shift;;
+    --enable-hard-errors) shift;;
+    # Explicitly terminate option list.
+    --) shift; break;;
+    # Shouldn't happen
+    *) echo "$0: invalid option/argument: '$1'" >&2; exit 2;;
+  esac
+  shift
+done
+st=0
+"$@" || st=$?
+rm -f "$log_file"
+case $st,$expect_failure in
+  0,no) echo "PASS: $test_name"; exit 0;;
+  1,no)  echo "FAIL: $test_name"; exit 1;;
+  0,yes) echo "XPASS: $test_name"; exit 1;;
+  1,yes) echo "XFAIL: $test_name"; exit 0;;
+  *) echo "UNEXPECTED OUTCOME: $test_name"; exit 99;;
+esac | tee "$log_file"
+END
+chmod a+x td
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+VERBOSE=yes $MAKE check
+test -f test-suite.log
+test -f sub1/test-suite.log
+test -f sub2/test-suite.log
+
+for dir in sub1 sub2; do
+  cd $dir
+  cp pass.test x1.test
+  cp x2.test pass.test
+  $MAKE check && { cat test-suite.log; Exit 1; }
+  cat test-suite.log
+  grep '^FAIL: pass\.test$' test-suite.log
+  grep '^XPASS: x1\.test$' test-suite.log
+  test `grep -c '^FAIL' test-suite.log` -eq 1
+  test `grep -c '^XPASS' test-suite.log` -eq 1
+  cd ..
+done
+
+:
diff --git a/tests/test-driver-custom.test b/tests/test-driver-custom.test
new file mode 100755
index 0000000..b6a33d2
--- /dev/null
+++ b/tests/test-driver-custom.test
@@ -0,0 +1,140 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Custom test drivers: per-extension test drivers.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+# Automake shouldn't match the '/test' in 'sub/test' as '.test' suffix.
+TESTS = 1.chk 2.test 3 4.c.chk 5.suf sub/test
+
+TEST_EXTENSIONS = .chk .test
+
+CHK_LOG_DRIVER = ./chk-wrapper
+TEST_LOG_DRIVER = $(SHELL) $(srcdir)/test-wrapper
+LOG_DRIVER = noext-wrapper
+
+AM_CHK_LOG_DRIVER_FLAGS = --am-chk
+CHK_LOG_DRIVER_FLAGS = --chk
+AM_TEST_LOG_DRIVER_FLAGS = -am-test
+TEST_LOG_DRIVER_FLAGS = -test
+AM_LOG_DRIVER_FLAGS = am
+LOG_DRIVER_FLAGS = _
+END
+
+mkdir sub bin
+PATH=`pwd`/bin$PATH_SEPARATOR$PATH; export PATH
+
+cat > wrapper.skel <<'END'
+#! /bin/sh
+set -e
+
+me=`echo "$0" | sed 's,^.*/,,'`
+if test -z "$me"; then
+  echo "$0: cannot determine program name" >&2
+  exit 99
+fi
+
+am_log_wflags='@am_log_wflags@'
+log_wflags='@log_wflags@'
+
+test_name=INVALID
+log_file=BAD.log
+extra_opts=
+while test $# -gt 0; do
+  case $1 in
+    --test-name) test_name=$2; shift;;
+    --log-file) log_file=$2; shift;;
+    # Ignored.
+    --expect-failure) shift;;
+    --color-tests) shift;;
+    --enable-hard-errors) shift;;
+    # Remembered in the same order they're passed in.
+    $am_log_wflags|$log_wflags) extra_opts="$extra_opts $1";;
+    # Explicitly terminate option list.
+    --) shift; break;;
+    # Shouldn't happen
+    *) echo "$0: invalid option/argument: '$1'" >&2; exit 2;;
+  esac
+  shift
+done
+
+echo "$me" "$test_name" $extra_opts > "$log_file"
+
+exec "$@"
+exit 127
+END
+
+sed -e 's|@am_log_wflags@|--am-chk|' \
+    -e 's|@log_wflags@|--chk|' \
+    < wrapper.skel > chk-wrapper
+
+sed -e 's|@am_log_wflags@|-am-test|' \
+    -e 's|@log_wflags@|-test|' \
+    < wrapper.skel > test-wrapper
+
+sed -e 's|@am_log_wflags@|am|' \
+    -e 's|@log_wflags@|_|' \
+    < wrapper.skel > bin/noext-wrapper
+
+# `test-wrapper' is deliberately not made executable.
+chmod a+x chk-wrapper bin/noext-wrapper
+
+# Not needed anymore.
+rm -f wrapper.skel
+
+cat > 1.chk << 'END'
+#! /bin/sh
+exit 0
+END
+chmod a+x 1.chk
+cp 1.chk 2.test
+cp 1.chk 3
+cp 1.chk 4.c.chk
+cp 1.chk 5.suf
+cp 1.chk sub/test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+$MAKE
+VERBOSE=yes $MAKE check
+ls -l . sub
+
+test ! -r BAD.log
+
+echo 'chk-wrapper 1.chk --am-chk --chk' > 1.exp
+echo 'test-wrapper 2.test -am-test -test' > 2.exp
+echo 'noext-wrapper 3 am _' > 3.exp
+echo 'chk-wrapper 4.c.chk --am-chk --chk' > 4.c.exp
+echo 'noext-wrapper 5.suf am _' > 5.suf.exp
+echo 'noext-wrapper sub/test am _' > sub/test.exp
+
+st=0
+for x in 1 2 3 4.c 5.suf sub/test; do
+  cat $x.log
+  diff $x.exp $x.log || st=1
+done
+
+Exit $st
diff --git a/tests/test-driver-fail.test b/tests/test-driver-fail.test
new file mode 100755
index 0000000..e1d2f9c
--- /dev/null
+++ b/tests/test-driver-fail.test
@@ -0,0 +1,65 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Custom test drivers: what happens when a test driver fails?  Well,
+# "make check" should at least fail too, and the test-suite.log
+# shouldn't be created.  Unfortunately, we cannot truly control also
+# the (non-)creation of individual test logs, since those are expected
+# to be created by the drivers themselves, and an ill-behaved driver
+# (like our dummy one in this test) might leave around a test log even
+# in case of internal failures.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in <<'END'
+AC_OUTPUT
+END
+
+cat > Makefile.am <<'END'
+TEST_LOG_DRIVER = ./oops
+TESTS = foo.test
+END
+
+cat > foo.test <<'END'
+#! /bin/sh
+exit 0
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+# The testsuite driver does not exist.
+$MAKE check && Exit 1
+test ! -f test-suite.log
+
+# The testsuite driver exists and create the test log files, but fails.
+
+cat > oops <<'END'
+#!/bin/sh
+: > foo.log
+echo 'Oops, I fail!' >&2
+exit 1
+END
+chmod a+x oops
+
+$MAKE check && Exit 1
+test ! -f test-suite.log
+
+:
-- 
1.7.2.3