[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Automake-commit] [SCM] GNU Automake branch, tap-testsuite-work, updated
|
From: |
Stefano Lattarini |
|
Subject: |
[Automake-commit] [SCM] GNU Automake branch, tap-testsuite-work, updated. v1.11-1098-g0cf750d |
|
Date: |
Fri, 05 Aug 2011 22:13:59 +0000 |
This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "GNU Automake".
http://git.sv.gnu.org/gitweb/?p=automake.git;a=commitdiff;h=0cf750d7143939e1118358557a19012a74fdb9d3
The branch, tap-testsuite-work has been updated
via 0cf750d7143939e1118358557a19012a74fdb9d3 (commit)
via 3027d901f4c4698241e074be5063df5b18fa7879 (commit)
via a3345817c8806193742307561787d74117f4b35f (commit)
via 144628d7033cdb098cf77fb94fb5204286adf10e (commit)
via ccf385b973b8eccce6eb38e3b6a35b8b6beded31 (commit)
via cf2e31d3b1f8e7f124c3194417685c0e206fa039 (commit)
via f5cb0635a837eb99591ac0356f242f0496f1ad0f (commit)
via a0d4d97a740412add8673379f7b36b2a40f6418f (commit)
via 4839ca9219b95a6155e4161438318ec0aca3aaef (commit)
via 5e3ae4e14a471a7c61ac75ffd6cd71659e81b07f (commit)
via bc5ba174ea24265b6970e9b747c2cf791f203dc2 (commit)
via 956c58ab490f79be1829ac4fc3a0da58a4b30973 (commit)
via 445674ff92cb0929f21627dec0ced4bd05a1c5c8 (commit)
from 0e886a6709d920b1f52f6cc80612db0b9d135cf9 (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
- Log -----------------------------------------------------------------
commit 0cf750d7143939e1118358557a19012a74fdb9d3
Author: Stefano Lattarini <address@hidden>
Date: Fri Aug 5 22:58:03 2011 +0200
testsuite: use custom TAP diagnostic in our own tests
This change allow us to easily and transparently avoid spurious
TAP diagnostic in our own testsuite, with the help of the new
`--diagnostic-string' option of our TAP test driver.
* tests/Makefile.am (TAP_LOG_DRIVER_FLAGS): Also pass option
`--diagnostic-string' with value `#%#' (necessarily obtained
through some shell indirections).
* tests/tap-functions.sh (diag_): Assume that the string denoting
TAP diagnostic is stored in the `$diag_string_' variable. Update
comments.
($diag_string_): Initialize to "#".
* tests/defs ($diag_string_): Re-define to "#%#".
(show_): Remove, it's obsolete now.
* add-missing.tap: Don't use `show_' anymore to avoid spurious
TAP diagnostic; `cat' should suffice now.
commit 3027d901f4c4698241e074be5063df5b18fa7879
Merge: 0e886a6 a334581
Author: Stefano Lattarini <address@hidden>
Date: Fri Aug 5 21:14:53 2011 +0200
Merge branch 'test-protocols' into tap-testsuite-work
* test-protocols:
tap: fix typo in TAP driver version message
tap docs: we don't support pragma or version directives (yet)
tap: new option to change the string designating TAP diagnostic
check: add small "synchronization" comment to `test-driver'
maintcheck: fix spurious failure w.r.t. use of bare "exit"
check: update comments to reflect recent heavy changes
check: small cleanups and refactorings in test harness and drivers
testsuite: use AM_TESTS_FD_REDIRECT where appropriate
docs: some fixlets in documentation on testsuites support
test harness: be aware of more metadata, simplify test drivers
tap: check that also a trailing TAP plan can hold a skip directive
-----------------------------------------------------------------------
Summary of changes:
ChangeLog | 163 ++++++++++++++++++++
automake.in | 6 -
doc/automake.texi | 103 +++++++++----
lib/Automake/tests/Makefile.in | 39 +++---
lib/am/check.am | 79 +++++------
lib/tap-driver | 63 ++++----
lib/test-driver | 43 ++----
tests/Makefile.am | 12 ++
tests/Makefile.in | 45 +++---
tests/add-missing.tap | 17 ++-
tests/check12.test | 28 +++-
tests/defs | 27 +---
tests/parallel-tests-harderror.test | 6 +-
tests/parallel-tests-interrupt.test | 14 +-
tests/parallel-tests-log-override-2.test | 5 +-
tests/parallel-tests-log-override-recheck.test | 30 ++--
tests/parallel-tests-unreadable.test | 4 +-
tests/tap-diagnostic-custom.test | 96 ++++++++++++
tests/tap-diagnostic.test | 2 +-
tests/tap-functions.sh | 10 +-
tests/tap-global-log.test | 2 +-
tests/tap-global-result.test | 46 ++++--
tests/tap-skip-whole-lastline.test | 59 +++++++
tests/test-log.test | 25 +++-
tests/test-metadata-global-result.test | 196 ++++++++++++++++++++++++
tests/test-metadata-results.test | 37 +++--
26 files changed, 866 insertions(+), 291 deletions(-)
create mode 100755 tests/tap-diagnostic-custom.test
create mode 100755 tests/tap-skip-whole-lastline.test
create mode 100755 tests/test-metadata-global-result.test
diff --git a/ChangeLog b/ChangeLog
index 0b46b4c..e8756c6 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,166 @@
+2011-08-05 Stefano Lattarini <address@hidden>
+
+ testsuite: use custom TAP diagnostic in our own tests
+ This change allow us to easily and transparently avoid spurious
+ TAP diagnostic in our own testsuite, with the help of the new
+ `--diagnostic-string' option of our TAP test driver.
+ * tests/Makefile.am (TAP_LOG_DRIVER_FLAGS): Also pass option
+ `--diagnostic-string' with value `#%#' (necessarily obtained
+ through some shell indirections).
+ * tests/tap-functions.sh (diag_): Assume that the string denoting
+ TAP diagnostic is stored in the `$diag_string_' variable. Update
+ comments.
+ ($diag_string_): Initialize to "#".
+ * tests/defs ($diag_string_): Re-define to "#%#".
+ (show_): Remove, it's obsolete now.
+ * add-missing.tap: Don't use `show_' anymore to avoid spurious
+ TAP diagnostic; `cat' should suffice now.
+
+2011-08-05 Stefano Lattarini <address@hidden>
+
+ tap: fix typo in TAP driver version message
+ * lib/tap-driver (anonymous subroutine printing the script version
+ message): Add missing trailing newline to the version message.
+
+2011-08-05 Stefano Lattarini <address@hidden>
+
+ tap docs: we don't support pragma or version directives (yet)
+ * doc/automake.texi (Incompatibilities with other TAP parsers
+ and drivers): Document that limitation. Normalize an unrelated
+ "TODO" item.
+
+2011-08-05 Stefano Lattarini <address@hidden>
+
+ tap: new option to change the string designating TAP diagnostic
+ Here we introduce a new option `--diagnostic-string' in our TAP
+ test driver, that allows the user to specify which string should
+ denote the beginning of a TAP diagnostic line. This change is
+ not gratuitous, nor result if over-engineering: it is motivated
+ by real issues that have emerged during the use of TAP in the
+ Automake's own testsuite (see the commit `v1.11-1082-g9b967c2'
+ "testsuite: yet more use of TAP, and related extensions").
+ * doc/automake.texi (Use TAP with Automake test harness): Document
+ the new option.
+ (Incompatibilities with other TAP parsers and drivers): Report it
+ as a potential source of incompatibility.
+ * lib/tap-driver ($diag_string): New global variable, defaulting
+ to "#", and whose value can be changed ...
+ (Getopt::Long::GetOptions): ... by the newly recognized option
+ `--diagnostic-string'.
+ (handle_tap_comment): Subroutine removed, some of its simple logic
+ inlined ...
+ (main): ... in here, where now ...
+ (extract_tap_comment): ... this new subroutine is used.
+ ($USAGE): Adjust.
+ * tests/tap-diagnostic.test: Make one check slightly stricter.
+ * tests/tap-diag-custom.test: New test.
+ * tests/Makefile.am (tap_other_tests): Add it.
+
+2011-08-05 Stefano Lattarini <address@hidden>
+
+ check: add small "synchronization" comment to `test-driver'
+ * lib/test-driver: Add comment to the code initializing ANSI color
+ escapes, telling to keep it in sync with the similar initialization
+ in `lib/am/check.am'.
+
+2011-08-05 Stefano Lattarini <address@hidden>
+
+ maintcheck: fix spurious failure w.r.t. use of bare "exit"
+ * tests/check12.test: Cosmetic changes to avoid triggering a
+ spurious failure of the `sc_tests_Exit_not_exit' maintainer
+ check.
+
+2011-08-05 Stefano Lattarini <address@hidden>
+
+ check: update comments to reflect recent heavy changes
+ * lib/am/check.am [%?PARALLEL_TESTS%]: Update the comments to
+ reflect the recent changes. Remove quite many comments that
+ were merely duplicating excerpts from the Automake manual.
+
+2011-08-05 Stefano Lattarini <address@hidden>
+
+ check: small cleanups and refactorings in test harness and drivers
+ * lib/tap-driver (yn): New subroutine, converts a boolean value to
+ a "yes" or "no" string.
+ (write_test_results): Use it, and related reformatting.
+ * lib/test-driver (trap): Also remove the `.trs' file on signals,
+ for extra safety.
+ (fatal): Remove this function, it's never used.
+ ($logfile, $trsfile): Renamed ...
+ ($log_file, $trs_file): ... to these, for clarity, and in order to
+ be more consistent with the `tap-driver' script.
+ Improve a couple of comments.
+ * automake.in (handle_tests): Don't define anymore the now-obsolete
+ make macro `$(TEST_LOGS_TMP)', nor add it to the list of files to
+ be removed upon "make mostlyclean".
+ * lib/am/check.am ($(TEST_SUITE_LOG)): New shell function `f_ok',
+ tells whether a path refers to an existing, regular, readable file.
+ Use it throughout.
+ (recheck): Be safer w.r.t. make implementation that run recipes
+ with `errexit' shell flag active.
+
+2011-08-05 Stefano Lattarini <address@hidden>
+
+ testsuite: use AM_TESTS_FD_REDIRECT where appropriate
+ * parallel-tests-interrupt.test: Use AM_TESTS_FD_REDIRECT, not
+ AM_TESTS_ENVIRONMENT, to define file descriptors' redirection
+ for "make check". Fix botched comment.
+
+2011-08-05 Stefano Lattarini <address@hidden>
+
+ docs: some fixlets in documentation on testsuites support
+ * doc/automake.texi (Script-based Testsuites): Rename node ...
+ (Scripts-based Testsuites): ... to this. Break overly long lines
+ in an example (were causing problems with PDF output).
+ (Parallel Test Harness): Don't intend paragraphs that come after
+ examples or bulleted list.
+ (Overview of Custom Test Drivers Support): Fix typo s/the the/the/.
+ (Declaring Custom Test Drivers): When giving an aside, prefer comma
+ to parentheses, as it disrupts the reading flow less.
+ (API for Custom Test Drivers): Remove a sentence that was basically
+ duplicated from the previous subsection.
+ (Command-line arguments for test drivers): Don't intend paragraph
+ coming after an itemized list. Clarify wording and fix grammaros.
+ Add a couple of cross-references to earlier explanations of `.log'
+ and `.trs' files.
+ (Links and external resources): Node renamed ...
+ (Links and external resources on TAP): ... to this, for clarity.
+ Slightly extend and clarify introductory sentence.
+
+2011-08-04 Stefano Lattarini <address@hidden>
+
+ test harness: be aware of more metadata, simplify test drivers
+ * lib/am/check.am ($(TEST_SUITE_LOG)): When producing the
+ global test log, take into account the new metadata field
+ `:global-test-result:, to write nicely formatted RST section
+ titles, instead of leaving this chore to the individual test
+ drivers.
+ (am_rst_section): Re-introduce this variable, as removed in
+ commit `v1.11-870-ga27c9c4'.
+ * lib/test-driver, lib/tap-driver: Write the global test result
+ as metadata in the `.trs' file, not as (part of) a formatted RST
+ subsection title in the `.log' file. Related simplifications;
+ in particular, get rid of temporary files usage.
+ * doc/automake.texi (Log files generation and test results
+ recording): Document the new metadata.
+ * tests/check12.test: Update.
+ * tests/parallel-tests-harderror.test: Likewise.
+ * tests/parallel-tests-interrupt.test: Likewise.
+ * tests/parallel-tests-log-override-2.test: Likewise.
+ * tests/parallel-tests-log-override-recheck.test: Likewise.
+ * tests/parallel-tests-unreadable.test: Likewise.
+ * tests/tap-global-result.test: Likewise.
+ * tests/test-metadata-results.test: Likewise.
+ * tests/test-log.test: Likewise, and another minor unrelated fix.
+ * tests/test-metadata-global-result.test: New test.
+ * tests/Makefile.am (TESTS): Update.
+
+2011-08-03 Stefano Lattarini <address@hidden>
+
+ tap: check that also a trailing TAP plan can hold a skip directive
+ * tests/tap-skip-whole-lastline.test: New test.
+ * tests/Makefile.am (tap_with_common_setup_tests): Add it.
+
2011-08-04 Stefano Lattarini <address@hidden>
testsuite: more correct names for a couple of tests
diff --git a/automake.in b/automake.in
index ad58f06..bccacf1 100644
--- a/automake.in
+++ b/automake.in
@@ -5118,12 +5118,6 @@ sub handle_tests
am__EXEEXT => $am_exeext);
}
}
-
- # FIXME: this is partly out-of-date w.r.t. the rest of the
- # FIXME: code now ... what is the best fix?
- define_variable ('TEST_LOGS_TMP', '$(TEST_LOGS:.log=.log-t)',
INTERNAL);
- $clean_files{'$(TEST_LOGS_TMP)'} = MOSTLY_CLEAN;
-
$clean_files{'$(TEST_LOGS)'} = MOSTLY_CLEAN;
$clean_files{'$(am__TEST_BASES:=.trs)'} = MOSTLY_CLEAN;
$clean_files{'$(TEST_SUITE_LOG)'} = MOSTLY_CLEAN;
diff --git a/doc/automake.texi b/doc/automake.texi
index 5eb40d5..c63742d 100644
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -326,7 +326,7 @@ Support for test suites
Simple Tests
-* Script-based Testsuites:: Automake-specific concepts and terminology
+* Scripts-based Testsuites:: Automake-specific concepts and terminology
* Serial Test Harness:: Older (and obsolescent) serial test harness
* Parallel Test Harness:: Generic concurrent test harness
@@ -335,7 +335,7 @@ Using the TAP test protocol
* Introduction to TAP::
* Use TAP with the Automake test harness::
* Incompatibilities with other TAP parsers and drivers::
-* Links and external resources::
+* Links and external resources on TAP::
Custom Test Drivers
@@ -8795,13 +8795,13 @@ terminology)?
@section Simple Tests
@menu
-* Script-based Testsuites:: Automake-specific concepts and terminology
+* Scripts-based Testsuites:: Automake-specific concepts and terminology
* Serial Test Harness:: Older (and obsolescent) serial test harness
* Parallel Test Harness:: Generic concurrent test harness
@end menu
address@hidden Script-based Testsuites
address@hidden Script-based Testsuites
address@hidden Scripts-based Testsuites
address@hidden Scripts-based Testsuites
If the special variable @code{TESTS} is defined, its value is taken to be
a list of programs or scripts to run in order to do the testing. Under
@@ -8931,9 +8931,9 @@ HP-UX's @command{/bin/sh},
@c Keep in sync with tests-environment-backcompat.test.
@example
AM_TESTS_ENVIRONMENT = \
-## Some environment initializations are kept in a separate shell file
-## `tests-env.sh', which can make it easier to also run tests from
-## the command line.
+## Some environment initializations are kept in a separate shell
+## file `tests-env.sh', which can make it easier to also run tests
+## from the command line.
. $(srcdir)/tests-env.sh; \
## On Solaris, prefer more POSIX-compliant versions of the standard
## tools by default.
@@ -8941,10 +8941,10 @@ AM_TESTS_ENVIRONMENT = \
PATH=/usr/xpg4/bin:$$PATH; export PATH; \
fi;
@c $$ restore font-lock
-## With this, the test scripts will be able to print diagnostic messages
-## to the original standard error stream, even if the test driver
-## redirects the stderr of the test scripts to a log file before executing
-## them.
+## With this, the test scripts will be able to print diagnostic
+## messages to the original standard error stream, even if the test
+## driver redirects the stderr of the test scripts to a log file
+## before executing them.
AM_TESTS_FD_REDIRECT = 9>&2
@end example
@@ -9129,6 +9129,7 @@ AM_LOG_FLAGS = -Mstrict -w
@trindex check-html
@vindex RST2HTML
@vindex TEST_SUITE_HTML
address@hidden
With @code{make check-html}, the log files may be converted from RST
(reStructuredText, see @uref{http://docutils.sourceforge.net/@/rst.html})
to HTML using @samp{RST2HTML}, which defaults to @command{rst2html} or
@@ -9204,6 +9205,7 @@ does the same as @code{recheck} but again converts the
resulting
@file{.log} file in HTML format, like the @code{check-html} target.
@end itemize
address@hidden
In order to guarantee an ordering between tests even with @code{make
address@hidden, dependencies between the corresponding @file{.log} files
may be specified through usual @command{make} dependencies. For example,
@@ -9295,7 +9297,7 @@ protocols are TAP and SubUnit).
It's very important to note that, even when using custom test drivers,
most of the infrastructure described in the previous section about the
-the parallel harness remains in place; this includes:
+parallel harness remains in place; this includes:
@itemize
@item
@@ -9361,9 +9363,9 @@ coexist.
The developer-reserved variable @code{AM_LOG_DRIVER_FLAGS} and the
user-reserved variable @code{LOG_DRIVER_FLAGS} can be used to define
-flags that will be passed to each invocation of @code{LOG_DRIVER}
-(with the user-defined flags obviously taking precedence over the
-developer-reserved ones). Similarly, for each extension @var{ext}
+flags that will be passed to each invocation of @code{LOG_DRIVER},
+with the user-defined flags obviously taking precedence over the
+developer-reserved ones. Similarly, for each extension @var{ext}
declared in @code{TEST_EXTENSIONS}, flags listed in
@address@hidden and
@address@hidden will be passed to
@@ -9378,9 +9380,7 @@ for new features or to satisfy additional portability
requirements.
The main characteristic of these APIs is that they are designed to share
as much infrastructure, semantics, and implementation details as possible
-with the parallel test harness and its default driver. So everything
-said in the previous section should apply here too, unless explicitly
-stated otherwise.
+with the parallel test harness and its default driver.
@menu
* Command-line arguments for test drivers::
@@ -9409,13 +9409,15 @@ suffix and a directory component (as in e.g.,
@file{sub/foo.test}), and is
mostly meant to be used in console reports about testsuite advancements and
results (@pxref{Testsuite progress output}).
@item address@hidden@var{PATH}.log}
-The @file{.log} file the test driver must create. If it has a directory
-component (as in e.g., @file{sub/foo.log}), the test harness will ensure
-that such directory exist @emph{before} the test driver is called.
+The @file{.log} file the test driver must create (@pxref{Basics of
+test metadata}). If it has a directory component (as in e.g.,
address@hidden/foo.log}), the test harness will ensure that such directory
+exists @emph{before} the test driver is called.
@item address@hidden@var{PATH}.trs}
-The @file{.trs} file the test driver must create. If it has a directory
-component (as in e.g., @file{sub/foo.trs}), the test harness will ensure
-that such directory exist @emph{before} the test driver is called.
+The @file{.trs} file the test driver must create (@pxref{Basics of
+test metadata}). If it has a directory component (as in e.g.,
address@hidden/foo.trs}), the test harness will ensure that such directory
+exists @emph{before} the test driver is called.
@item address@hidden|address@hidden
Whether the console output should be colorized or not (@pxref{Simple
tests and color-tests}, to learn when this option gets activated and
@@ -9431,9 +9433,10 @@ conventions in use.
Explicitly terminate the list of options.
@end table
-The first of the remaining arguments passed to the test driver is the
-program to be run, and the other arguments are command-line options
-and arguments for this program.
address@hidden
+The first non-option argument passed to the test driver is the program to
+be run, and all the following ones are command-line options and arguments
+for this program.
Note that the exact semantics attached to the @option{--color-tests},
@option{--expect-failure} and @option{--enable-hard-errors} options are
@@ -9508,6 +9511,19 @@ just a waste of space in normal situations, e.g., when a
test script is
successful. What happens when two or more @code{:copy-in-global-log:}
fields are present in the same @file{.trs} file is undefined behaviour.
address@hidden Keep in sync with 'test-metadata-global-result.test'.
address@hidden @code{:test-global-result:}
address@hidden :test-global-result:
address@hidden reStructuredText field, @code{:test-global-result:}
+This is used to declare the "global result" of the script. Currently,
+the value of this field is needed only to be reported (more or less
+verbatim) in the generated global log file @code{$(TEST_SUITE_LOG)},
+so it's quite free-form. For example, a test script which run 10 test
+cases, 6 of which pass and 4 of which are skipped, could reasonably have
+a @code{PASS/SKIP} value for this field, while a test script which run
+19 successful tests and one failed test could have an @code{ALMOST
+PASSED} value. What happens when two or more @code{:test-global-result:}
+fields are present in the same @file{.trs} file is undefined behaviour.
@end table
@noindent
@@ -9576,7 +9592,7 @@ harness should remain untouched, and continue to work
correctly.
* Introduction to TAP::
* Use TAP with the Automake test harness::
* Incompatibilities with other TAP parsers and drivers::
-* Links and external resources::
+* Links and external resources on TAP::
@end menu
@node Introduction to TAP
@@ -9651,6 +9667,17 @@ harness to get confused if anything that appears on
standard error
looks like a test result.
@item --no-merge
Revert the effects of @option{--merge}.
address@hidden address@hidden
+Change the string that introduces TAP diagnostic from the default value
+of address@hidden'' to @address@hidden This can be useful if your
+TAP-based test scripts produce verbose output on which they have limited
+control (because, say, the output comes by other tools invoked in the
+scripts), and it might contain text that gets spuriously interpreted as
+TAP diagnostic: such an issue can be solved by redefining the string that
+activates TAP diagnostic to a value you know won't appear by chance in
+the tests' output. Note however that this feature is non-standard, as
+the ``official'' TAP protocol does not allow for such a customization; so
+don't use it if you can avoid it.
@end table
@noindent
@@ -9745,15 +9772,25 @@ the test script it occurs into. This doesn't follows
TAP specifications,
but on the other hand it maximizes compatibility (and code sharing) with
the ``hard error'' concept of the default @option{parallel-tests} driver.
@item
address@hidden: there's surely something else ...
+The @code{version} and @code{pragma} directives are not supported.
address@hidden
+The @option{--diagnostic-string} option of out driver allows to modify
+the string that introduces TAP diagnostic from the default value
+of address@hidden''. The standard TAP protocol has currently no way to
+allow this, so if you use it your diagnostic will be lost to more
+compliant tools like @command{prove} and @code{Test::Harness}
address@hidden
+And there are probably some other small and yet undiscovered
+incompatibilities, especially in corner cases or with rare usages.
@end itemize
address@hidden Links and external resources
address@hidden Links and external resources
address@hidden Links and external resources on TAP
address@hidden Links and external resources on TAP
@noindent
Here are some links to more extensive official or third-party
-documentation and resources:
+documentation and resources about the TAP protocol and related
+tools and libraries.
@itemize @bullet
@item
@uref{http://search.cpan.org/~petdance/Test-Harness/lib/Test/Harness/TAP.pod,
diff --git a/lib/Automake/tests/Makefile.in b/lib/Automake/tests/Makefile.in
index f375e3c..d401d1d 100644
--- a/lib/Automake/tests/Makefile.in
+++ b/lib/Automake/tests/Makefile.in
@@ -116,8 +116,9 @@ am__nobase_list = $(am__nobase_strip_setup); \
am__base_list = \
sed '$$!N;$$!N;$$!N;$$!N;$$!N;$$!N;$$!N;s/\n/ /g' | \
sed '$$!N;$$!N;$$!N;$$!N;s/\n/ /g'
-# Restructured Text title.
+# Restructured Text title and section.
am__rst_title = sed 's/.*/ & /;h;s/./=/g;p;x;p;g;p;s/.*//'
+am__rst_section = sed 'p;s/./=/g;p;g'
# Solaris 10 'make', and several other traditional 'make' implementations,
# pass "-e" to $(SHELL), and POSIX 2008 even requires this. Work around it
# by disabling -e (using the XSI extension "set +e") if it's set.
@@ -173,7 +174,6 @@ am__test_logs1 = $(TESTS:=.log)
TEST_LOGS = $(am__test_logs1:.pl.log=.log)
PL_LOG_DRIVER = $(SHELL) $(top_srcdir)/lib/test-driver
PL_LOG_COMPILE = $(PL_LOG_COMPILER) $(AM_PL_LOG_FLAGS) $(PL_LOG_FLAGS)
-TEST_LOGS_TMP = $(TEST_LOGS:.log=.log-t)
DISTFILES = $(DIST_COMMON) $(DIST_SOURCES) $(TEXINFOS) $(EXTRA_DIST)
ACLOCAL = @ACLOCAL@
AMTAR = @AMTAR@
@@ -353,12 +353,11 @@ cscope cscopelist:
$(TEST_SUITE_LOG): $(TEST_LOGS)
@$(am__sh_e_setup); $(am__tty_colors); \
+ f_ok () { test -f "$$1" && test -r "$$1"; }; \
bases='$(am__TEST_BASES)'; \
ws='[ ]'; \
redo_bases=`for b in $$bases; do \
- test -f $$b.trs && test -r $$b.trs \
- && test -f $$b.log && test -r $$b.log \
- || echo $$b; \
+ f_ok $$b.trs && f_ok $$b.log || echo $$b; \
done`; \
if test -n "$$redo_bases"; then \
redo_logs=`for i in $$redo_bases; do echo $$i.log; done`; \
@@ -369,10 +368,10 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
st=0; \
for b in $$redo_bases; do \
for e in trs log; do \
- if test ! -f $$b.$$e || test ! -r $$b.$$e; then \
+ f_ok $$b.$$e || { \
echo "fatal: making $@: failed to create $$b.$$e" >&2; \
st=1; \
- fi; \
+ }; \
done; \
done; \
test $$st -eq 0 || exit 1; \
@@ -427,15 +426,20 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
echo ".. contents:: :depth: 2"; \
echo; \
for b in $$bases; do \
- if grep "^$$ws*:copy-in-global-log:$$ws*no$$ws*$$" $$b.trs \
- >/dev/null; then :; \
- elif test ! -r $$b.log; then \
- echo "ERROR: cannot read $$b.log" >&2; \
- echo && echo "WARNING: could not read $$b.log!"; \
- else \
- echo && cat $$b.log; \
- fi; \
- done; \
+ if grep "^$$ws*:copy-in-global-log:$$ws*no$$ws*$$" $$b.trs \
+ >/dev/null; then continue; \
+ fi; \
+ glob_res=`sed -n -e "s/$$ws*$$//" \
+ -e "s/^$$ws*:global-test-result:$$ws*//p" \
+ $$b.trs`; \
+ test -n "$$glob_res" || glob_res=RUN; \
+ echo "$$glob_res: $$b" | $(am__rst_section); \
+ if test ! -r $$b.log; then \
+ echo "fatal: making $@: $$b.log is unreadable" >&2; \
+ exit 1; \
+ fi; \
+ cat $$b.log; echo; \
+ done; \
} >$(TEST_SUITE_LOG).tmp; \
mv $(TEST_SUITE_LOG).tmp $(TEST_SUITE_LOG); \
if $$success; then \
@@ -498,7 +502,7 @@ recheck recheck-html:
target=`echo $@ | sed 's,^re,,'`; \
bases='$(am__TEST_BASES)'; \
list=`for b in $$bases; do \
- test ! -f $$b.trs && test ! -f $$b.log && continue; \
+ test -f $$b.trs || test -f $$b.log || continue; \
grep "^$$ws*:recheck:$$ws*no$$ws*$$" $$b.trs \
>/dev/null 2>&1 || echo $$b.log; \
done | tr '\012\015' ' '`; \
@@ -566,7 +570,6 @@ install-strip:
fi
mostlyclean-generic:
-test -z "$(TEST_LOGS)" || rm -f $(TEST_LOGS)
- -test -z "$(TEST_LOGS_TMP)" || rm -f $(TEST_LOGS_TMP)
-test -z "$(TEST_SUITE_HTML)" || rm -f $(TEST_SUITE_HTML)
-test -z "$(TEST_SUITE_LOG)" || rm -f $(TEST_SUITE_LOG)
-test -z "$(am__TEST_BASES:=.trs)" || rm -f $(am__TEST_BASES:=.trs)
diff --git a/lib/am/check.am b/lib/am/check.am
index c406e9f..8340850 100644
--- a/lib/am/check.am
+++ b/lib/am/check.am
@@ -49,34 +49,16 @@ include inst-vars.am
## New parallel test driver.
##
-## This code is adapted from check.mk which was originally
-## written at EPITA/LRDE, further developed at Gostai, then made
-## its way from GNU coreutils to end up, largely rewritten, in
-## Automake.
-##
-## It provides special support for "unit tests", that is to say,
-## tests that (once run) no longer need to be re-compiled and
-## re-run at each "make check", unless their sources changed. To
-## enable unit-test supports, set RECHECK_LOGS to empty. In such a
-## setting, that heavily relies on correct dependencies, its users may
-## prefer to define EXTRA_PROGRAMS instead of check_PROGRAMS, because
-## it allows intertwined compilation and execution of the tests.
-## Sometimes this helps catching errors earlier (you don't have to
-## wait for all the tests to be compiled).
-##
-## Define TEST_SUITE_LOG to be the name of the global log to create.
-## Define TEST_LOGS to the set of logs to include in it. It defaults
-## to $(TESTS), with `.test' and address@hidden@' removed, and `'.log'
-## appended.
-##
-## In addition to the magic "exit 77 means SKIP" feature (which was
-## imported from automake), there is a magic "exit 99 means ERROR" feature
-## which is useful if you need to issue a hard error no matter whether the
-## test is XFAIL or not. You can disable this feature by setting the
-## variable DISABLE_HARD_ERRORS to a nonempty value.
+## The first version of the code here was adapted from check.mk, which was
+## originally written at EPITA/LRDE, further developed at Gostai, then made
+## its way from GNU coreutils to end up, largely rewritten, in Automake.
+## The current version is an heavy rewrite of that, to allow for support
+## of more test metadata, and the use of custom test derivers and protocols
+## (among them, TAP).
-# Restructured Text title.
+# Restructured Text title and section.
am__rst_title = sed 's/.*/ & /;h;s/./=/g;p;x;p;g;p;s/.*//'
+am__rst_section = sed 'p;s/./=/g;p;g'
# Solaris 10 'make', and several other traditional 'make' implementations,
# pass "-e" to $(SHELL), and POSIX 2008 even requires this. Work around it
@@ -140,6 +122,9 @@ am__stealth_MAKE = $(MAKE)
$(TEST_SUITE_LOG): $(TEST_LOGS)
@$(am__sh_e_setup); $(am__tty_colors); \
+## Helper shell function, tells whether a path refers to an existing,
+## regular, readable file.
+ f_ok () { test -f "$$1" && test -r "$$1"; }; \
bases='$(am__TEST_BASES)'; \
ws='[ ]'; \
## We need to ensures that all the required `.trs' and `.log' files will
@@ -148,9 +133,7 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
## the `.log' files are readable, and worse, they don't ensure that the
## `.trs' files even exist.
redo_bases=`for b in $$bases; do \
- test -f $$b.trs && test -r $$b.trs \
- && test -f $$b.log && test -r $$b.log \
- || echo $$b; \
+ f_ok $$b.trs && f_ok $$b.log || echo $$b; \
done`; \
if test -n "$$redo_bases"; then \
## Uh-oh, either some `.log' files were unreadable, or some `.trs' files
@@ -167,10 +150,10 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
st=0; \
for b in $$redo_bases; do \
for e in trs log; do \
- if test ! -f $$b.$$e || test ! -r $$b.$$e; then \
+ f_ok $$b.$$e || { \
echo "fatal: making $@: failed to create $$b.$$e" >&2; \
st=1; \
- fi; \
+ }; \
done; \
done; \
test $$st -eq 0 || exit 1; \
@@ -240,18 +223,26 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
echo; \
for b in $$bases; do \
## FIXME: one fork per test -- this is horrendously inefficient!
- if grep "^$$ws*:copy-in-global-log:$$ws*no$$ws*$$" $$b.trs \
- >/dev/null; then :; \
-## If we cannot read the .log file of a test ...
- elif test ! -r $$b.log; then \
-## ... print an error message to the console ...
- echo "ERROR: cannot read $$b.log" >&2; \
-## ... and a warning in the `test-suite.log' file.
- echo && echo "WARNING: could not read $$b.log!"; \
- else \
- echo && cat $$b.log; \
- fi; \
- done; \
+ if grep "^$$ws*:copy-in-global-log:$$ws*no$$ws*$$" $$b.trs \
+ >/dev/null; then continue; \
+ fi; \
+## Get the declared "global result" of the test.
+## FIXME: yet another one fork per test here!
+ glob_res=`sed -n -e "s/$$ws*$$//" \
+ -e "s/^$$ws*:global-test-result:$$ws*//p" \
+ $$b.trs`; \
+## If no global result is explicitly declared, we'll merely mark the
+## test as "RUN" in the global test log.
+ test -n "$$glob_res" || glob_res=RUN; \
+## Write the name and result of the test as an RST section title.
+ echo "$$glob_res: $$b" | $(am__rst_section); \
+## If we should have remade any unreadable `.log', above.
+ if test ! -r $$b.log; then \
+ echo "fatal: making $@: $$b.log is unreadable" >&2; \
+ exit 1; \
+ fi; \
+ cat $$b.log; echo; \
+ done; \
} >$(TEST_SUITE_LOG).tmp; \
mv $(TEST_SUITE_LOG).tmp $(TEST_SUITE_LOG); \
## Emit the test summary on the console.
@@ -353,7 +344,7 @@ recheck recheck-html:
list=`for b in $$bases; do \
## Skip tests that haven't been run, but recover gracefully from deleted
## `.trs' files.
- test ! -f $$b.trs && test ! -f $$b.log && continue; \
+ test -f $$b.trs || test -f $$b.log || continue; \
## FIXME: one fork per test -- this is horrendously inefficient!
grep "^$$ws*:recheck:$$ws*no$$ws*$$" $$b.trs \
>/dev/null 2>&1 || echo $$b.log; \
diff --git a/lib/tap-driver b/lib/tap-driver
index 3facdfd..91edd1e 100755
--- a/lib/tap-driver
+++ b/lib/tap-driver
@@ -17,8 +17,9 @@ my $USAGE = <<'END';
Usage:
tap-driver --test-name=NAME --log-file=PATH --trs-file=PATH
[--expect-failure={yes|no}] [--color-tests={yes|no}]
- [--enable-hard-errors={yes|no}] [--merge|--no-merge]
- [--ignore-exit] [--comments|--no-comments] [--] TEST-COMMAND
+ [--enable-hard-errors={yes|no}] [--ignore-exit]
+ [--diagnostic-string=STRING] [--merge|--no-merge]
+ [--comments|--no-comments] [--] TEST-COMMAND
The `--test-name' and `--log-file' options are mandatory.
END
@@ -63,16 +64,18 @@ my %cfg = (
my $test_script_name = undef;
my $log_file = undef;
my $trs_file = undef;
+my $diag_string = "#";
Getopt::Long::GetOptions (
'help' => sub { print $HELP; exit 0; },
- 'version' => sub { print "$ME $VERSION"; exit 0; },
+ 'version' => sub { print "$ME $VERSION\n"; exit 0; },
'test-name=s' => \$test_script_name,
'log-file=s' => \$log_file,
'trs-file=s' => \$trs_file,
'color-tests=s' => \&bool_opt,
'expect-failure=s' => \&bool_opt,
'enable-hard-errors=s' => \&bool_opt,
+ 'diagnostic-string=s' => \$diag_string,
'comments' => sub { $cfg{"comments"} = 1; },
'no-comments' => sub { $cfg{"comments"} = 0; },
'merge' => sub { $cfg{"merge"} = 1; },
@@ -101,6 +104,13 @@ sub bool_opt ($$)
}
}
+# Convert a boolean to a "yes"/"no" string.
+sub yn ($)
+{
+ my $bool = shift;
+ return $bool ? "yes" : "no";
+}
+
TAP_PEEKING :
{
my @tap_lines = ();
@@ -169,10 +179,9 @@ TEST_RESULTS :
sub write_test_results ()
{
open RES, ">", $trs_file or die "opening $trs_file: $!\n";
- print RES ":recheck: " .
- (must_recheck ? "yes" : "no") . "\n";
- print RES ":copy-in-global-log: " .
- (copy_in_global_log ? "yes" : "no") . "\n";
+ print RES ":global-test-result: " . get_global_test_result . "\n";
+ print RES ":recheck: " . yn (must_recheck) . "\n";
+ print RES ":copy-in-global-log: " . yn (copy_in_global_log) . "\n";
foreach my $result (get_test_results)
{
print RES ":test-result: $result\n";
@@ -185,10 +194,10 @@ sub start (@)
# Redirect stderr and stdout to a temporary log file. Save the
# original stdout stream, since we need it to print testsuite
# progress output.
+ open LOG, ">", $log_file or die "opening $log_file: $!\n";
open OLDOUT, ">&STDOUT" or die "duplicating stdout: $!\n";
- open TMP, ">$log_file-t" or die "opening $log_file-t: $!\n";
- open STDOUT, ">&TMP" or die "redirecting stdout: $!\n";
- open STDERR, ">&TMP" or die "redirecting stderr: $!\n";
+ open STDOUT, ">&LOG" or die "redirecting stdout: $!\n";
+ open STDERR, ">&LOG" or die "redirecting stderr: $!\n";
$parser = TAP::Parser->new ({ exec => address@hidden, merge => $cfg{merge}
});
$parser->ignore_exit(1) if $cfg{"ignore-exit"};
}
@@ -196,20 +205,7 @@ sub start (@)
sub finish ()
{
write_test_results;
-
- open LOG, ">", $log_file or die "opening $log_file: $!\n";
- my $global_result = get_global_test_result;
- my $global_result_line = "$global_result: $test_script_name";
- print LOG "$global_result_line\n";
- print LOG "=" x length ($global_result_line) . "\n";
- print LOG "\n";
-
- close TMP or die "closing $log_file-t: $!\n";
- # FIXME: remove this hack!
- my $test_output = `cat $log_file-t && rm -f $log_file-t`;
- print LOG $test_output;
close LOG or die "closing $log_file: $!\n";
-
exit 0;
}
@@ -355,11 +351,15 @@ sub handle_tap_bailout ($)
finish;
}
-sub handle_tap_comment ($)
+sub extract_tap_comment ($)
{
- return unless $cfg{comments};
- my $comment = $_[0]->comment;
- report "#", "$comment" if length $comment;
+ local $_ = shift;
+ if (/^\Q$diag_string\E(.*)$/o)
+ {
+ (my $comment = $1) =~ s/(?:^\s*|\s*$)//;
+ return $comment;
+ }
+ return "";
}
sub main (@)
@@ -379,14 +379,15 @@ sub main (@)
{
handle_tap_test ($cur);
}
- elsif ($cur->is_comment)
- {
- handle_tap_comment ($cur);
- }
elsif ($cur->is_bailout)
{
handle_tap_bailout ($cur);
}
+ elsif ($cfg{comments})
+ {
+ my $comment = extract_tap_comment ($cur->raw);
+ report "#", "$comment" if length $comment;
+ }
}
if (!$plan_seen)
{
diff --git a/lib/test-driver b/lib/test-driver
index 472ba49..8522a49 100755
--- a/lib/test-driver
+++ b/lib/test-driver
@@ -1,7 +1,7 @@
#! /bin/sh
# test-driver - basic driver script for the `parallel-tests' mode.
-scriptversion=2011-08-01.21; # UTC
+scriptversion=2011-08-04.22; # UTC
# Copyright (C) 2011 Free Software Foundation, Inc.
#
@@ -31,12 +31,6 @@ scriptversion=2011-08-01.21; # UTC
# helps a lot in preventing typo-related bugs.
set -u
-fatal ()
-{
- echo "$0: fatal: $*" >&2
- exit 1
-}
-
usage_error ()
{
echo "$0: $*" >&2
@@ -55,14 +49,11 @@ The \`--test-name' and \`--log-file' options are mandatory.
END
}
-# Restructured Text section.
-rst_section () { sed 'p;s/./=/g;p;g'; }
-
# TODO: better error handling in option parsing (in particular, ensure
-# TODO: $logfile, $trsfile and $test_name are defined).
+# TODO: $log_file, $trs_file and $test_name are defined).
test_name= # Used for reporting.
-logfile= # Where to save the output of the test script.
-trsfile= # Where to save the result(s) the test script.
+log_file= # Where to save the output of the test script.
+trs_file= # Where to save the metadata of the test run.
expect_failure=no
color_tests=no
enable_hard_errors=yes
@@ -71,8 +62,8 @@ while test $# -gt 0; do
--help) print_usage; exit $?;;
--version) echo "test-driver $scriptversion"; exit $?;;
--test-name) test_name=$2; shift;;
- --log-file) logfile=$2; shift;;
- --trs-file) trsfile=$2; shift;;
+ --log-file) log_file=$2; shift;;
+ --trs-file) trs_file=$2; shift;;
--color-tests) color_tests=$2; shift;;
--expect-failure) expect_failure=$2; shift;;
--enable-hard-errors) enable_hard_errors=$2; shift;;
@@ -83,6 +74,7 @@ while test $# -gt 0; do
done
if test $color_tests = yes; then
+ # Keep this in sync with `lib/am/check.am:$(am__tty_colors)'.
red='[0;31m' # Red.
grn='[0;32m' # Green.
lgn='[1;32m' # Light green.
@@ -93,16 +85,14 @@ else
red= grn= lgn= blu= mgn= std=
fi
-tmpfile=$logfile-t
-do_exit='rm -f $tmpfile; (exit $st); exit $st'
+do_exit='rm -f $log_file $trs_file; (exit $st); exit $st'
trap "st=129; $do_exit" 1
trap "st=130; $do_exit" 2
trap "st=141; $do_exit" 13
trap "st=143; $do_exit" 15
-rm -f $tmpfile
# Test script is run here.
-"$@" >$tmpfile 2>&1
+"$@" >$log_file 2>&1
estatus=$?
if test $enable_hard_errors = no && test $estatus -eq 99; then
estatus=1
@@ -121,17 +111,10 @@ esac
echo "${col}${res}${std}: $test_name"
# Register the test result, and other relevant metadata.
-echo ":test-result: $res (exit status: $estatus)" > $trsfile
-echo ":recheck: $recheck" >> $trsfile
-echo ":copy-in-global-log: $gcopy" >> $trsfile
-
-# Now write log file.
-{
- echo "$res: $test_name (exit: $estatus)" | rst_section
- echo
- cat $tmpfile
-} > $logfile
-rm -f $tmpfile
+echo ":test-result: $res" > $trs_file
+echo ":global-test-result: $res" >> $trs_file
+echo ":recheck: $recheck" >> $trs_file
+echo ":copy-in-global-log: $gcopy" >> $trs_file
# Local Variables:
# mode: shell-script
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 56c6795..425112b 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -19,6 +19,15 @@
TEST_EXTENSIONS = .test .tap
TAP_LOG_DRIVER = $(PERL) $(top_srcdir)/lib/tap-driver
TAP_LOG_DRIVER_FLAGS = --merge --comments
+## Avoid spurious TAP diagnostic. We use an awful indirection with a
+## shell command substitution here, since the string needs to contain
+## `#' characters, and these can't be escaped portably in a make macro
+## definition.
+## FIXME: this is quite inefficient though, as it adds one extra fork
+## FIXME: per TAP test script; this is OK for the moment, since we have
+## FIXME: few such scripts, but might become an issue in the future.
+## Keep this in sync with the definition of $diag_string_ in tests/defs.
+TAP_LOG_DRIVER_FLAGS += --diagnostic-string `printf '\043%%\043\n'`
MAINTAINERCLEANFILES =
EXTRA_DIST = ChangeLog-old
@@ -807,6 +816,7 @@ test-driver-trs-suffix-registered.test \
test-driver-fail.test \
test-log.test \
test-metadata-global-log.test \
+test-metadata-global-result.test \
test-metadata-recheck.test \
test-metadata-results.test \
test-missing.test \
@@ -1220,6 +1230,7 @@ tap-realtime.test \
tap-recheck-logs.test \
tap-skip-whole-whitespace.test \
tap-skip-whole.test \
+tap-skip-whole-lastline.test \
tap-todo-skip-together.test \
tap-todo-skip-whitespace.test \
tap-todo-skip.test \
@@ -1240,6 +1251,7 @@ tap_other_tests = \
tap-common-setup.test \
tap-bad-prog.tap \
tap-basic.test \
+tap-diagnostic-custom.test \
tap-doc.test \
tap-doc2.test \
tap-more.test \
diff --git a/tests/Makefile.in b/tests/Makefile.in
index 4ac3220..d371452 100644
--- a/tests/Makefile.in
+++ b/tests/Makefile.in
@@ -121,8 +121,9 @@ am__nobase_list = $(am__nobase_strip_setup); \
am__base_list = \
sed '$$!N;$$!N;$$!N;$$!N;$$!N;$$!N;$$!N;s/\n/ /g' | \
sed '$$!N;$$!N;$$!N;$$!N;s/\n/ /g'
-# Restructured Text title.
+# Restructured Text title and section.
am__rst_title = sed 's/.*/ & /;h;s/./=/g;p;x;p;g;p;s/.*//'
+am__rst_section = sed 'p;s/./=/g;p;g'
# Solaris 10 'make', and several other traditional 'make' implementations,
# pass "-e" to $(SHELL), and POSIX 2008 even requires this. Work around it
# by disabling -e (using the XSI extension "set +e") if it's set.
@@ -198,7 +199,6 @@ TEST_LOGS = $(am__test_logs6:.shtst.log=.log)
SHTST_LOG_DRIVER = $(SHELL) $(top_srcdir)/lib/test-driver
SHTST_LOG_COMPILE = $(SHTST_LOG_COMPILER) $(AM_SHTST_LOG_FLAGS) \
$(SHTST_LOG_FLAGS)
-TEST_LOGS_TMP = $(TEST_LOGS:.log=.log-t)
DISTFILES = $(DIST_COMMON) $(DIST_SOURCES) $(TEXINFOS) $(EXTRA_DIST)
ACLOCAL = @ACLOCAL@
AMTAR = @AMTAR@
@@ -300,7 +300,8 @@ top_builddir = @top_builddir@
top_srcdir = @top_srcdir@
TEST_EXTENSIONS = .test .tap .ptest .instspc .depmod .shtst
TAP_LOG_DRIVER = $(PERL) $(top_srcdir)/lib/tap-driver
-TAP_LOG_DRIVER_FLAGS = --merge --comments
+TAP_LOG_DRIVER_FLAGS = --merge --comments --diagnostic-string `printf \
+ '\043%%\043\n'`
MAINTAINERCLEANFILES =
EXTRA_DIST = ChangeLog-old gen-parallel-tests parallel-tests.sh \
instspc-tests.sh depmod-tests.sh gen-config-shell-tests \
@@ -1083,6 +1084,7 @@ test-driver-trs-suffix-registered.test \
test-driver-fail.test \
test-log.test \
test-metadata-global-log.test \
+test-metadata-global-result.test \
test-metadata-recheck.test \
test-metadata-results.test \
test-missing.test \
@@ -1480,6 +1482,7 @@ tap-realtime.test \
tap-recheck-logs.test \
tap-skip-whole-whitespace.test \
tap-skip-whole.test \
+tap-skip-whole-lastline.test \
tap-todo-skip-together.test \
tap-todo-skip-whitespace.test \
tap-todo-skip.test \
@@ -1497,6 +1500,7 @@ tap_other_tests = \
tap-common-setup.test \
tap-bad-prog.tap \
tap-basic.test \
+tap-diagnostic-custom.test \
tap-doc.test \
tap-doc2.test \
tap-more.test \
@@ -1567,12 +1571,11 @@ cscope cscopelist:
$(TEST_SUITE_LOG): $(TEST_LOGS)
@$(am__sh_e_setup); $(am__tty_colors); \
+ f_ok () { test -f "$$1" && test -r "$$1"; }; \
bases='$(am__TEST_BASES)'; \
ws='[ ]'; \
redo_bases=`for b in $$bases; do \
- test -f $$b.trs && test -r $$b.trs \
- && test -f $$b.log && test -r $$b.log \
- || echo $$b; \
+ f_ok $$b.trs && f_ok $$b.log || echo $$b; \
done`; \
if test -n "$$redo_bases"; then \
redo_logs=`for i in $$redo_bases; do echo $$i.log; done`; \
@@ -1583,10 +1586,10 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
st=0; \
for b in $$redo_bases; do \
for e in trs log; do \
- if test ! -f $$b.$$e || test ! -r $$b.$$e; then \
+ f_ok $$b.$$e || { \
echo "fatal: making $@: failed to create $$b.$$e" >&2; \
st=1; \
- fi; \
+ }; \
done; \
done; \
test $$st -eq 0 || exit 1; \
@@ -1641,15 +1644,20 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
echo ".. contents:: :depth: 2"; \
echo; \
for b in $$bases; do \
- if grep "^$$ws*:copy-in-global-log:$$ws*no$$ws*$$" $$b.trs \
- >/dev/null; then :; \
- elif test ! -r $$b.log; then \
- echo "ERROR: cannot read $$b.log" >&2; \
- echo && echo "WARNING: could not read $$b.log!"; \
- else \
- echo && cat $$b.log; \
- fi; \
- done; \
+ if grep "^$$ws*:copy-in-global-log:$$ws*no$$ws*$$" $$b.trs \
+ >/dev/null; then continue; \
+ fi; \
+ glob_res=`sed -n -e "s/$$ws*$$//" \
+ -e "s/^$$ws*:global-test-result:$$ws*//p" \
+ $$b.trs`; \
+ test -n "$$glob_res" || glob_res=RUN; \
+ echo "$$glob_res: $$b" | $(am__rst_section); \
+ if test ! -r $$b.log; then \
+ echo "fatal: making $@: $$b.log is unreadable" >&2; \
+ exit 1; \
+ fi; \
+ cat $$b.log; echo; \
+ done; \
} >$(TEST_SUITE_LOG).tmp; \
mv $(TEST_SUITE_LOG).tmp $(TEST_SUITE_LOG); \
if $$success; then \
@@ -1712,7 +1720,7 @@ recheck recheck-html:
target=`echo $@ | sed 's,^re,,'`; \
bases='$(am__TEST_BASES)'; \
list=`for b in $$bases; do \
- test ! -f $$b.trs && test ! -f $$b.log && continue; \
+ test -f $$b.trs || test -f $$b.log || continue; \
grep "^$$ws*:recheck:$$ws*no$$ws*$$" $$b.trs \
>/dev/null 2>&1 || echo $$b.log; \
done | tr '\012\015' ' '`; \
@@ -1805,7 +1813,6 @@ install-strip:
fi
mostlyclean-generic:
-test -z "$(TEST_LOGS)" || rm -f $(TEST_LOGS)
- -test -z "$(TEST_LOGS_TMP)" || rm -f $(TEST_LOGS_TMP)
-test -z "$(TEST_SUITE_HTML)" || rm -f $(TEST_SUITE_HTML)
-test -z "$(TEST_SUITE_LOG)" || rm -f $(TEST_SUITE_LOG)
-test -z "$(am__TEST_BASES:=.trs)" || rm -f $(am__TEST_BASES:=.trs)
diff --git a/tests/add-missing.tap b/tests/add-missing.tap
index 809b30f..4c6a624 100755
--- a/tests/add-missing.tap
+++ b/tests/add-missing.tap
@@ -54,8 +54,8 @@ $ACLOCAL || framework_failure_ "cannot pre-compute aclocal.m4"
rm -rf install-sh missing depcomp configure.in autom4te*.cache
mv aclocal.m4 aclocal.stub
-show_ configure.stub
-show_ aclocal.stub
+cat configure.stub
+cat aclocal.stub
# This is hacky and ugly and complex, but allow us to organize our tests
# below in a more "declarative fashion". All in all, a good trade-off.
@@ -111,15 +111,18 @@ check_ ()
if test $run_aclocal = yes; then
if $ACLOCAL; then
ok_ "[$name] aclocal.m4 rebuilt"
- show_ aclocal.m4
+ echo == aclocal.m4 ==
+ cat aclocal.m4
else
not_ok_ "[$name] aclocal failure, aclocal.m4 not rebuilt"
fi
else
cp ../aclocal.stub aclocal.m4
fi
- show_ Makefile.am
- show_ configure.in
+ echo == Makefile.am ==
+ cat Makefile.am
+ echo == configure.in ==
+ cat configure.in
echo Expected files: $files
mkdir "$build_aux"
cd ..
@@ -173,8 +176,8 @@ check_ ()
all_files="install-sh missing $files"
for f in $all_files; do echo $f; done | sort | uniq > files.exp
(cd $build_aux && ls) | sort > files.got
- show_ files.exp
- show_ files.got
+ cat files.exp
+ cat files.got
command_ok_ \
"$pfx all and only expected files installed" \
$am_diff files.exp files.got
diff --git a/tests/check12.test b/tests/check12.test
index 4be4b80..e36a94d 100755
--- a/tests/check12.test
+++ b/tests/check12.test
@@ -34,10 +34,12 @@ END
cat > a.test << 'END'
#!/bin/sh
+echo a.test: exit status: ${A_EXIT_STATUS-0}
exit ${A_EXIT_STATUS-0}
END
cat > b.test << 'END'
#!/bin/sh
+echo b.test: exit status: ${B_EXIT_STATUS-0}
exit ${B_EXIT_STATUS-0}
END
chmod +x a.test b.test
@@ -135,9 +137,9 @@ for vpath in : false; do
test -f spanner.sum
# This checks will be run only by the autogenerated `check12-p.test'.
if test x"$parallel_tests" = x"yes"; then
- cat test-suite.log
- grep '^PASS: a\.test (exit: 0)' a.log
- grep '^PASS: b\.test (exit: 0)' b.log
+ test -f test-suite.log
+ test -f a.log
+ test -f b.log
else
:
fi
@@ -170,9 +172,13 @@ for vpath in : false; do
# This checks will be run only by the autogenerated `check12-p.test'.
if test x"$parallel_tests" = x"yes"; then
cat test-suite.log
- grep '^PASS: a\.test (exit: 0)' a.log
- grep '^FAIL: b\.test (exit: 1)' b.log
- grep '^FAIL: b\.test (exit: 1)' test-suite.log
+ cat a.log
+ cat b.log
+ grep '^a\.test: exit status: 0$' a.log
+ grep '^b\.test: exit status: 1$' b.log
+ grep '^FAIL: b$' test-suite.log
+ grep '^b\.test: exit status: 1$' test-suite.log
+ grep '^a\.test' test-suite.log && Exit 1
else :; fi
CHECKLOCAL_EXIT_STATUS=1 $MAKE check && Exit 1
@@ -189,9 +195,13 @@ for vpath in : false; do
# This checks will be run only by the autogenerated `check12-p.test'.
if test x"$parallel_tests" = x"yes"; then
cat test-suite.log
- grep '^PASS: a\.test (exit: 0)' a.log
- grep '^FAIL: b\.test (exit: 23)' b.log
- grep '^FAIL: b\.test (exit: 23)' test-suite.log
+ cat a.log
+ cat b.log
+ grep '^a\.test: exit status: 0$' a.log
+ grep '^b\.test: exit status: 23$' b.log
+ grep '^FAIL: b$' test-suite.log
+ grep '^b\.test: exit status: 23$' test-suite.log
+ grep '^a\.test' test-suite.log && Exit 1
else :; fi
grep 'check-local failed :-(' local.log
diff --git a/tests/defs b/tests/defs
index 3ce92fd..c468d5f 100644
--- a/tests/defs
+++ b/tests/defs
@@ -183,6 +183,10 @@ else
fi
unset funcs_file_
+# Avoid spurious TAP diagnostic. Keep this in sync with the definition
+# of $(TAP_LOG_DRIVER_FLAGS) in `tests/Makefile.am'.
+diag_string_="#%#"
+
# cross_compiling
# ---------------
# Tell whether we are cross-compiling. This is especially useful to skip
@@ -202,29 +206,6 @@ is_newest ()
test -z "$is_newest_files"
}
-# show_ [--prepend STRING] [FILES...]
-# ----------------------------
-# Show the content of each of the given files that is not empty,
-# prepending every one of its lines with the given STRING (defaulting
-# to " | "). STRING cannot contain the "#" or "\" character.
-# This function is not just eye-candy and syntactic sugar: in tests
-# that use the TAP protocols, a line beginning with a "#" character
-# would be interpreted by the TAP driver as a diagnostic line, and
-# possibly displayed on console (with the risk of cluttering precious
-# screen space with noise).
-show_ ()
-{
- set +x
- p_=" | "
- case $1 in -p|--prepend) p_=$2; shift 2;; esac
- for f_ in ${1+"$@"}; do
- test -s "$f_" || { echo "== $f_ is empty"; continue; }
- echo "== $f_ =="
- sed "s#^#$p_#" "$f_"
- done
- set -x
-}
-
# AUTOMAKE_run [-e STATUS] [-d DESCRIPTION] [--] [AUTOMAKE-ARGS...]
# -----------------------------------------------------------------
# Run automake with AUTOMAKE-ARGS, and fail if it doesn't exit with
diff --git a/tests/parallel-tests-harderror.test
b/tests/parallel-tests-harderror.test
index fc3b804..f49b79b 100755
--- a/tests/parallel-tests-harderror.test
+++ b/tests/parallel-tests-harderror.test
@@ -67,13 +67,13 @@ DISABLE_HARD_ERRORS=x $MAKE check
$MAKE check DISABLE_HARD_ERRORS='' && Exit 1
cat test-suite.log
-grep '^ERROR: foo\.test .*exit.*99' test-suite.log
+grep '^ERROR: foo$' test-suite.log
cd sub
# The `-e' is wanted here.
DISABLE_HARD_ERRORS='' $MAKE -e check && Exit 1
cat test-suite.log
-grep '^ERROR: bar\.test .*exit.*99' test-suite.log
+grep '^ERROR: bar$' test-suite.log
cd ..
# Check the distributions.
@@ -88,7 +88,7 @@ $AUTOMAKE Makefile
./config.status Makefile
VERBOSE=yes $MAKE check && Exit 1
grep '^FAIL' test-suite.log && Exit 1
-grep '^ERROR: bar\.test .*exit.*99' sub/test-suite.log
+grep '^ERROR: bar$' sub/test-suite.log
echo 'DISABLE_HARD_ERRORS = zardoz' >> sub/Makefile
VERBOSE=yes $MAKE check
diff --git a/tests/parallel-tests-interrupt.test
b/tests/parallel-tests-interrupt.test
index 1c5d9ce..ed40412 100755
--- a/tests/parallel-tests-interrupt.test
+++ b/tests/parallel-tests-interrupt.test
@@ -30,8 +30,8 @@ cat > Makefile.am << 'END'
TESTS = foo.test
## Provide more debugging info.
TEST_LOG_COMPILER = $(SHELL) -x
-## Rut required by foo.test; see below.
-AM_TESTS_ENVIRONMENT = 9>&2
+## Required by foo.test; see below.
+AM_TESTS_FD_REDIRECT = 9>&2
END
# This is hacky and ugly, but has the great advantage of avoiding us a lot
@@ -47,8 +47,8 @@ cat > foo.test << 'END'
exec 2>&9
echo "foo is starting to run"
ls -l >&2
-cat foo.log-t >&2 || : > fail
-grep '^foo is starting to run$' foo.log-t >&2 || : > fail
+cat foo.log >&2 || : > fail
+grep '^foo is starting to run$' foo.log >&2 || : > fail
cat pid >&2 || : > fail
kill -$signum `cat pid` || : > fail
END
@@ -65,17 +65,15 @@ $AUTOMAKE
trapped_signals='1 2 13 15'
for signum in $trapped_signals; do
- rm -f pid fail *.log *.log-t
+ rm -f pid fail *.log
env signum=$signum $MAKE check && { ls -l; Exit 1; }
ls -l
# These files shouldn't exist, but in case they do, their content might
# provide helpful information about the causes of the failure(s).
- cat foo.log-t || :
cat foo.log || :
cat test-suite.log || :
test -f fail && Exit 1
- test -f foo.log-t && Exit 1
- test -f foo.log && Exit 1
+ ls | $EGREP 'foo.*\.(log|tmp)' && Exit 1
done
:
diff --git a/tests/parallel-tests-log-override-2.test
b/tests/parallel-tests-log-override-2.test
index a0997a3..ef37a2c 100755
--- a/tests/parallel-tests-log-override-2.test
+++ b/tests/parallel-tests-log-override-2.test
@@ -75,9 +75,10 @@ do
test ! -f fail.log
grep '^PASS: pass\.test$' stdout
grep '^SKIP: skip\.test$' stdout
- $FGREP 'skip.test' partial.log
+ $FGREP 'SKIP: skip' partial.log
$FGREP '% skipped test %' partial.log
- $EGREP '(pass2|skip2|fail)\.test' stdout partial.log && Exit 1
+ $EGREP '(pass2|skip2|fail)\.test' stdout && Exit 1
+ $EGREP '(pass2|skip2|fail)' partial.log && Exit 1
rm -f *.log
done
diff --git a/tests/parallel-tests-log-override-recheck.test
b/tests/parallel-tests-log-override-recheck.test
index da768ef..7956930 100755
--- a/tests/parallel-tests-log-override-recheck.test
+++ b/tests/parallel-tests-log-override-recheck.test
@@ -25,22 +25,22 @@ AC_OUTPUT
END
cat > Makefile.am <<'END'
-TESTS = foo.test bar.test baz.test
+TESTS = foofoo.test barbar.test bazbaz.test
END
-cat > foo.test <<'END'
+cat > foofoo.test <<'END'
#! /bin/sh
echo "this is $0"
exit 0
END
-cat > bar.test <<'END'
+cat > barbar.test <<'END'
#! /bin/sh
echo "this is $0"
exit 99
END
-cat > baz.test <<'END'
+cat > bazbaz.test <<'END'
#! /bin/sh
echo "this is $0"
exit ${BAZ_EXIT_STATUS-1}
@@ -65,9 +65,9 @@ cat stdout
ls -l
count_test_results total=2 pass=0 fail=1 skip=0 xfail=0 xpass=0 error=1
for x in stdout my.log; do
- $FGREP foo.test $x && Exit 1
- $FGREP bar.test $x
- $FGREP baz.test $x
+ $FGREP foofoo $x && Exit 1
+ $FGREP barbar $x
+ $FGREP bazbaz $x
done
chmod a-rw my.log
@@ -76,15 +76,15 @@ BAZ_EXIT_STATUS=0 TEST_SUITE_LOG=my2.log $MAKE -e recheck
>stdout \
cat stdout
ls -l
count_test_results total=2 pass=1 fail=0 skip=0 xfail=0 xpass=0 error=1
-$FGREP foo.test stdout && Exit 1
-$FGREP bar.test stdout
-$FGREP baz.test stdout
-$FGREP foo.test my2.log && Exit 1
-$FGREP bar.test my2.log
-$FGREP baz.test my2.log && Exit 1
+$FGREP foofoo stdout && Exit 1
+$FGREP barbar stdout
+$FGREP bazbaz stdout
+$FGREP foofoo my2.log && Exit 1
+$FGREP barbar my2.log
+$FGREP bazbaz my2.log && Exit 1
chmod u+r test-suite.log my.log
-$FGREP baz.test test-suite.log
-$FGREP baz.test my.log
+$FGREP bazbaz test-suite.log
+$FGREP bazbaz my.log
:
diff --git a/tests/parallel-tests-unreadable.test
b/tests/parallel-tests-unreadable.test
index 47edc9a..6192a13 100755
--- a/tests/parallel-tests-unreadable.test
+++ b/tests/parallel-tests-unreadable.test
@@ -67,9 +67,9 @@ for files in \
grep '^:test-result: PASS' foo.trs
grep '^barbarbar$' bar.log
grep '^:test-result: SKIP' bar.trs
- grep '^SKIP: bar.test' test-suite.log
+ grep '^SKIP: bar' test-suite.log
grep '^barbarbar$' test-suite.log
- $EGREP 'foo\.test|foofoofoo' test-suite.log && Exit 1
+ $EGREP ':.*foo|foofoofoo' test-suite.log && Exit 1
done
:
diff --git a/tests/tap-diagnostic-custom.test b/tests/tap-diagnostic-custom.test
new file mode 100755
index 0000000..d965229
--- /dev/null
+++ b/tests/tap-diagnostic-custom.test
@@ -0,0 +1,96 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - option '--diagnostic-string' to customize the string introducing
+# TAP diagnostics
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+my_log_driver = $(PERL) $(srcdir)/tap-driver
+my_log_compiler = cat
+TEST_EXTENSIONS =
+TESTS =
+END
+
+: > later.mk
+
+# Quoting our comments below is an hack required to keep
+# comments near the things they refer to.
+i=0
+for string in \
+'## A letter' \
+ a \
+'## A number' \
+ 1023 \
+'## A non-alphabetic character' \
+ @ \
+'## Some metacharacters (we need to repeat the "$" for make)' \
+ '^>;&*"|$$' \
+'## A whitespace character' \
+ " " \
+'## A tab character' \
+ "$tab" \
+'## A string with more whitespace' \
+ " ${tab}a b${tab} c" \
+'## Note the we do not have the empty string here. We prefer to' \
+'## leave its behaviour in this context undefined for the moment.'
+do
+ case $string in '##'*) continue;; esac
+ i=`expr $i + 1`
+ unindent >> Makefile.am << END
+ TEST_EXTENSIONS += .t$i
+ TESTS += foo$i.t$i
+ T${i}_LOG_COMPILER = \$(my_log_compiler)
+ T${i}_LOG_DRIVER = \$(my_log_driver)
+ AM_T${i}_LOG_DRIVER_FLAGS = \
+ --comments \
+ --diagnostic-string '$string'
+END
+ unindent > foo$i.t$i <<END
+ 1..1
+ ok 1
+ $string blah blah $i
+END
+ echo "AM_T${i}_LOG_DRIVER_FLAGS = --no-comments" >> later.mk
+done
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=$i pass=$i fail=0 xpass=0 xfail=0 skip=0 error=0
+
+cat later.mk >> Makefile
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+$FGREP 'blah blah' stdout && Exit 1
+
+:
diff --git a/tests/tap-diagnostic.test b/tests/tap-diagnostic.test
index 73ab45e..3c7da79 100755
--- a/tests/tap-diagnostic.test
+++ b/tests/tap-diagnostic.test
@@ -117,7 +117,7 @@ count_test_results total=1 pass=1 fail=0 xpass=0 xfail=0
skip=0 error=0
grep "^# all.test:${ws0p}foo$" stdout
grep "^# all.test:${ws0p}bar${ws0p}$" stdout
grep "^# all.test:${ws1p}zardoz${ws0p}$" stdout
-grep "^# all.test:${ws1p}foo${ws1p}bar${ws1p}baz${ws0p}$" stdout
+grep "^# all.test:${ws1p}foo bar${tab}baz${ws0p}$" stdout
test `grep -c '^# all\.test:' stdout` -eq 4
diff --git a/tests/tap-functions.sh b/tests/tap-functions.sh
index 0104d39..d87a32c 100644
--- a/tests/tap-functions.sh
+++ b/tests/tap-functions.sh
@@ -63,12 +63,18 @@ have_tap_plan_=no
# diag_ [EXPLANATION]
# ------------------
-# Report the given text as TAP diagnostic.
+# Report the given text as TAP diagnostic. Assumes the string denoting
+# TAP diagnostic lines is stored in the `$diag_string_' variable; this is
+# done to allow better interplay with TAP drivers that allow such a string
+# to be configured.
diag_ ()
{
- test $# -eq 0 || echo "# $*"
+ test $# -eq 0 || echo "$diag_string_ $*"
}
+# Used by the `diag_' function above. User-overridable.
+diag_string_="#"
+
# warn_ [EXPLANATION]
# ------------------
# Give a warning (using TAP diagnostic).
diff --git a/tests/tap-global-log.test b/tests/tap-global-log.test
index a4f6428..d343581 100755
--- a/tests/tap-global-log.test
+++ b/tests/tap-global-log.test
@@ -106,7 +106,7 @@ END
TESTS="`echo *.test`" $MAKE -e check || :
cat test-suite.log
-grep 'ok\.test|not seen' test-suite.log && Exit 1
+grep ':.*ok|not seen' test-suite.log && Exit 1
for s in skip todo fail xpass bail error; do
$FGREP "::$s::" test-suite.log
diff --git a/tests/tap-global-result.test b/tests/tap-global-result.test
index c49fae4..32e2401 100755
--- a/tests/tap-global-result.test
+++ b/tests/tap-global-result.test
@@ -79,26 +79,42 @@ END
# TODO: add scripts with TAP errors (multiple plans, out-of-order
# tests, etc).
-TESTS="`echo *.test`" $MAKE -e check >stdout && { cat stdout; Exit 1; }
+tests=`echo *.test` # Also required later.
+
+TESTS="$tests" $MAKE -e check >stdout && { cat stdout; Exit 1; }
cat stdout
+# Dirty tricks required here.
for tst in ok skip skipall fail fail2 xpass xpass2 error error2 \
hodgepodge hodgepodge-all; do
- sed -e 2q $tst.log > $tst.res
+ echo :copy-in-global-log: yes >> $tst.trs
done
-cat *.res # For debugging.
-
-grep '^PASS:' ok.res
-grep '^SKIP:' skip.res
-grep '^SKIP:' skipall.res
-grep '^FAIL:' fail.res
-grep '^FAIL:' fail2.res
-grep '^FAIL:' xpass.res
-grep '^FAIL:' xpass2.res
-grep '^ERROR:' error.res
-grep '^ERROR:' error2.res
-grep '^ERROR:' hodgepodge.res
-grep '^ERROR:' hodgepodge-all.res
+rm -f test-suite.log
+TESTS="$tests" $MAKE -e test-suite.log && Exit 1
+cat test-suite.log
+
+have_rst_section ()
+{
+ eqeq=`echo "$1" | sed 's/./=/g'`
+ # Assume $1 contains no RE metacharacters.
+ sed -n "/^$1$/,/^$eqeq$/p" test-suite.log > got
+ (echo "$1" && echo "$eqeq") > exp
+ cat exp
+ cat got
+ diff exp got
+}
+
+have_rst_section 'PASS: ok'
+have_rst_section 'SKIP: skip'
+have_rst_section 'SKIP: skipall'
+have_rst_section 'FAIL: fail'
+have_rst_section 'FAIL: fail2'
+have_rst_section 'FAIL: xpass'
+have_rst_section 'FAIL: xpass2'
+have_rst_section 'ERROR: error'
+have_rst_section 'ERROR: error2'
+have_rst_section 'ERROR: hodgepodge'
+have_rst_section 'ERROR: hodgepodge-all'
:
diff --git a/tests/tap-skip-whole-lastline.test
b/tests/tap-skip-whole-lastline.test
new file mode 100755
index 0000000..25697e0
--- /dev/null
+++ b/tests/tap-skip-whole-lastline.test
@@ -0,0 +1,59 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - the special plan format to skip all the tests in a script can also
+# be used on the last line, even if preceded by non-TAP output, TAP
+# diagnostic, and even non-failing TAP test results.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+. "$testsrcdir"/tap-setup.sh || fatal_ "sourcing tap-setup.sh"
+
+echo AM_TEST_LOG_DRIVER_FLAGS = --comments >> Makefile
+
+cat > foo.test <<END
+some non-TAP text, will be copied in the global log
+1..0 # SKIP from the last line
+END
+
+cat > bar.test <<END
+# some TAP diagnostic, will go to console
+1..0
+END
+
+cat > baz.test <<END
+ok 1 - seen also if the whole test is skipped!
+1..0 # SKIP
+END
+
+st=0
+TESTS='foo.test bar.test baz.test' $MAKE -e check >stdout || st=$?
+cat stdout
+cat test-suite.log
+test $st -eq 0
+
+grep '^SKIP: foo\.test .* from the last line$' stdout
+grep '^SKIP: bar\.test$' stdout
+grep '^SKIP: baz\.test$' stdout
+grep '^# bar\.test: some TAP diagnostic, will go to console$' stdout
+grep '^some non-TAP text, will be copied in the global log$' test-suite.log
+grep '^PASS: baz\.test 1 - seen also if the whole test is skipped!$' stdout
+test `grep -c ': .*\.test' stdout` -eq 4
+count_test_results total=4 pass=1 fail=0 xpass=0 xfail=0 skip=3 error=0
+
+:
diff --git a/tests/test-log.test b/tests/test-log.test
index 1385386..c25cb20 100755
--- a/tests/test-log.test
+++ b/tests/test-log.test
@@ -50,7 +50,7 @@ cat > skip.test <<END
#! /bin/sh
echo "$pmarker skip $pmarker"
echo "# $cmarker skip $cmarker"
-exit 1
+exit 77
END
cat > xfail.test <<END
@@ -111,14 +111,25 @@ done
test `$FGREP -c "$pmarker" my.log` -eq 5
test `$FGREP -c "$cmarker" my.log` -eq 5
+have_rst_section ()
+{
+ eqeq=`echo "$1" | sed 's/./=/g'`
+ # Assume $1 contains no RE metacharacters.
+ sed -n "/^$1$/,/^$eqeq$/p" $2 > got
+ (echo "$1" && echo "$eqeq") > exp
+ cat exp
+ cat got
+ diff exp got
+}
+
# Passed test scripts shouldn't be mentioned in the global log.
-$EGREP '(^pass|[^x]pass)\.test' my.log && Exit 1
+$EGREP ':.*[^x]pass' my.log && Exit 1
# But failing (expectedly or not) and skipped ones should.
-$FGREP 'xfail.test' my.log
-$FGREP 'skip.test' my.log
-$FGREP 'fail.test' my.log
-$FGREP 'xpass.test' my.log
-$FGREP 'error.test' my.log
+have_rst_section 'SKIP: skip' my.log
+have_rst_section 'FAIL: fail' my.log
+have_rst_section 'XFAIL: xfail' my.log
+have_rst_section 'XPASS: xpass' my.log
+have_rst_section 'ERROR: error' my.log
touch error2.log test-suite.log global.log
TEST_SUITE_LOG=my.log $MAKE -e mostlyclean
diff --git a/tests/test-metadata-global-result.test
b/tests/test-metadata-global-result.test
new file mode 100755
index 0000000..d20b249
--- /dev/null
+++ b/tests/test-metadata-global-result.test
@@ -0,0 +1,196 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Parallel testsuite harness: check APIs for the registering the
+# "global test result" in `*.trs' files, as documented in the automake
+# manual.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_EXTENSIONS = .test .x
+TEST_LOG_DRIVER = ./dummy-driver
+X_LOG_DRIVER = ./dummy-driver
+TESTS = foo.test zar-doz.test
+END
+
+cat > dummy-driver <<'END'
+#! /bin/sh
+set -e; set -u
+while test $# -gt 0; do
+ case $1 in
+ --log-file) log_file=$2; shift;;
+ --trs-file) trs_file=$2; shift;;
+ --test-name) test_name=$2; shift;;
+ --expect-failure|--color-tests|--enable-hard-errors) shift;;
+ --) shift; break;;
+ *) echo "$0: invalid option/argument: '$1'" >&2; exit 2;;
+ esac
+ shift
+done
+echo logloglog > $log_file
+cp $1 $trs_file
+END
+chmod a+x dummy-driver
+
+# Do this in a subroutine to avoid quoting problem in the backticked
+# command substitution below.
+get_escaped_line()
+{
+ sed -e 's,[$^/\\\.],\\&,g' -e 1q "$@"
+}
+
+have_result ()
+{
+ cat > exp; echo >> exp; echo logloglog >> exp
+ eline=`get_escaped_line exp`
+ sed -n -e "/^$eline$/,/^logloglog$/p" test-suite.log > got
+ cat exp; cat got
+ diff exp got
+}
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+: Basic checks.
+
+echo :global-test-result: PASS > foo.test
+echo :global-test-result: ERROR > zar-doz.x
+
+$MAKE check
+cat test-suite.log
+
+have_result <<END
+PASS: foo
+=========
+END
+
+have_result <<END
+ERROR: zar-doz
+==============
+END
+
+: Try usage documented in the manual.
+
+echo :global-test-result: PASS/SKIP > foo.test
+echo :global-test-result: ALMOST PASSED > zar-doz.x
+
+$MAKE check
+cat test-suite.log
+
+have_result <<END
+PASS/SKIP: foo
+==============
+END
+
+have_result <<END
+ALMOST PASSED: zar-doz
+======================
+END
+
+: Fields ':test-result:' does not interfere with the global test result.
+
+cat > foo.test << 'END'
+:test-result: FAIL
+:global-test-result: PASS
+:test-result: ERROR
+END
+
+cat > zar-doz.x << 'END'
+:global-test-result: FAIL
+:test-result: SKIP
+:test-result: XFAIL
+END
+
+$MAKE check && Exit 1
+cat test-suite.log
+
+have_result <<END
+PASS: foo
+=========
+END
+
+have_result <<END
+FAIL: zar-doz
+=============
+END
+
+: What happens when ':global-test-result:' is absent.
+
+cat > foo.test << 'END'
+:test-result: PASS
+:test-result: ERROR
+END
+: > zar-doz.x
+
+$MAKE check && Exit 1
+cat test-suite.log
+
+have_result <<END
+RUN: foo
+========
+END
+
+have_result <<END
+RUN: zar-doz
+============
+END
+
+# Leading and trailing whitespace gets eaten/normalized.
+
+echo ":global-test-result:SKIP${tab} ${tab}${tab}" > foo.test
+echo ":global-test-result:${tab} ${tab}XFAIL ${tab} " > zar-doz.x
+
+$MAKE check
+cat test-suite.log
+
+have_result <<END
+SKIP: foo
+=========
+END
+
+have_result <<END
+XFAIL: zar-doz
+==============
+END
+
+# Whitespaces before and after `:global-test-result:' are handled OK.
+
+echo " $tab:global-test-result:PASS" > foo.test
+echo "${tab}${tab}:global-test-result:${tab} ${tab}SKIP" > zar-doz.x
+
+$MAKE check
+cat test-suite.log
+
+have_result <<END
+PASS: foo
+=========
+END
+
+have_result <<END
+SKIP: zar-doz
+=============
+END
+
+:
diff --git a/tests/test-metadata-results.test b/tests/test-metadata-results.test
index effa488..923f88f 100755
--- a/tests/test-metadata-results.test
+++ b/tests/test-metadata-results.test
@@ -84,36 +84,43 @@ $AUTOMAKE
./configure
-# Basic checks. Also that that "old-style" directives with format
-# "RESULT: test-name" are be ignored now.
+# Basic checks. Also that that `:global-test-result:' fields and
+# "old-style" directives with format "RESULT: test-name" are now ignored.
+
+: > foo.test
+echo blah blah blah > bar.test
+mk_check
+count_test_results total=0 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
+
+cat > foo.test <<END
+:test-global-result: PASS
+:test-result: FAIL
+END
+cat > bar.test <<END
+:test-result: SKIP
+:test-global-result: ERROR
+END
+mk_check && Exit 1
+count_test_results total=2 pass=0 fail=1 xpass=0 xfail=0 skip=1 error=0
cat > foo.test <<END
FAIL: foo.test
:test-result: PASS
+:test-global-result: XPASS
END
echo ERROR: bar.test > bar.test
mk_check
count_test_results total=1 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=0
cat > foo.test <<END
-PASS: foo.test
-:test-result: FAIL
-END
-echo SKIP: bar.test > bar.test
-: > bar.test
-mk_check && Exit 1
-count_test_results total=1 pass=0 fail=1 xpass=0 xfail=0 skip=0 error=0
-
-cat > foo.test <<END
-PASS: foo.test
+:test-global-result: SKIP
:test-result: FAIL
END
cat > bar.test <<END
-ERROR: foo.test
-:test-result: SKIP
+:test-global-result: PASS
END
mk_check && Exit 1
-count_test_results total=2 pass=0 fail=1 xpass=0 xfail=0 skip=1 error=0
+count_test_results total=1 pass=0 fail=1 xpass=0 xfail=0 skip=0 error=0
cat > foo.test <<END
:test-result: XFAIL
hooks/post-receive
--
GNU Automake
| [Prev in Thread] |
Current Thread |
[Next in Thread] |
- [Automake-commit] [SCM] GNU Automake branch, tap-testsuite-work, updated. v1.11-1098-g0cf750d,
Stefano Lattarini <=