[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Automake-commit] [SCM] GNU Automake branch, testsuite-work, updated. v1
|
From: |
Stefano Lattarini |
|
Subject: |
[Automake-commit] [SCM] GNU Automake branch, testsuite-work, updated. v1.11-1199-g66dc614 |
|
Date: |
Sun, 14 Aug 2011 16:23:36 +0000 |
This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "GNU Automake".
http://git.sv.gnu.org/gitweb/?p=automake.git;a=commitdiff;h=66dc614311f2010c6346a66a41dff14c1c0a8a65
The branch, testsuite-work has been updated
via 66dc614311f2010c6346a66a41dff14c1c0a8a65 (commit)
via f5c5453039bd46bc54023c1f0434c167624aba5b (commit)
via 08ddfe12f5bb387c4b1adc9e0950cada3ef0cbe0 (commit)
via e55b4ab196beaca28331d5bd2da69bc36b2f3f90 (commit)
via 38f12a3085be683b598be7d10de37eba0b629f28 (commit)
via 0d8a113dd070db481d0070a48e6577a12d8d7149 (commit)
via 726540faa1a5136004eb41d5b70f61e8aa2b795f (commit)
via 836ca56434e2d63b9ad8324a0b551f789878c3f6 (commit)
via a3e7e1baa3bb9bb95ecd754a1bb3e4806d9ffecc (commit)
via 7afca964324add2466d851aa07c3c650f624ed78 (commit)
via 9e5967708814027981d910e6fed4f7789eba23f8 (commit)
via c1d0de60cd864a0bf27df370a9d01640944ed8e2 (commit)
via e7ca7771e57e0d02c9dedf0e3bfc5128b397d7f2 (commit)
via 52a130494d747f224f4d6cc7a487942179857135 (commit)
via bf69289c8b6f060de7946fe455c12c2fbdddd053 (commit)
via 320685333ede68b51e2c720941b4f90010268760 (commit)
via d4a752a353c3fd15dbe7cb5d2b5e01996546d75c (commit)
via cc485aba6babbe23c0b67eb28eb6181f18fedb98 (commit)
via d35cee2a6c394e13b81ebef29fe53eea4e2b533d (commit)
via b2d6b49c50859a4b0773f6e20756e2ece20f092c (commit)
via 7fff2880a5e284f17cd6f0d869b0addeef7e1f55 (commit)
via 61873d53fea7a1a341bb7234d8428d5aba9ebf3b (commit)
via a7cbc8a792ee1a33ab1cedd1d89237487b2c6782 (commit)
via 1ebdc2c1b9a75420dd7518116a1e70b1ad6d93fc (commit)
via 89dca9e1858f3734c651a2e49134665dcf8fa606 (commit)
from a956d98634008022ff2541b8b9beebae1e025def (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
- Log -----------------------------------------------------------------
commit 66dc614311f2010c6346a66a41dff14c1c0a8a65
Merge: 08ddfe1 f5c5453
Author: Stefano Lattarini <address@hidden>
Date: Sun Aug 14 17:12:37 2011 +0200
Merge branch 'test-protocols' into testsuite-work
* test-protocols:
coverage: missing tap plan and non-zero exit status
testsuite: fix an unportable use of sed
testsuite: avoid spurious failure if rst2html program is missing
fix: two recent ChangeLog entries
tap: improve diagnosing and reporting of plan mismatches
tap: log all TAP stream, even after a "Bail out!"
coverage: TAP diagnostics after "Bail out!" aren't reported
coverage: TAP errors after a "Bail out!" directive aren't reported
testsuite: reorganize tests on TAP plans
coverage: more TAP driver consistency w.r.t. 'prove'
tap: improve some comments in the TAP driver
tap: non-zero exit status after "Bail out!" should not be reported
tap: a "plan with SKIP" after test results is an error
test harness: avoid possible fork bomb
coverage: possible infinite recursion in the test harness
coverage: possible infinite recursion in the test harness
test defs: yet more environment cleanup
commit 08ddfe12f5bb387c4b1adc9e0950cada3ef0cbe0
Author: Stefano Lattarini <address@hidden>
Date: Sun Aug 14 15:18:31 2011 +0200
self tests: some minor refactoring
* tests/self-check-exit.tap: Disable the `errexit' shell flag;
this allow to remove some boilerplate idioms and to make some
code slightly less cumbersome.
* tests/self-check-me.tap: Likewise.
commit e55b4ab196beaca28331d5bd2da69bc36b2f3f90
Author: Stefano Lattarini <address@hidden>
Date: Sun Aug 14 14:46:53 2011 +0200
maintcheck: look for TAP-based tests without TAP plan
* Makefile.am (sc_tests_tap_plan): New maintainer check.
(syntax_check_rules): Add it.
* tests/suffix10.tap: Declare a TAP plan.
* tests/add-missing.tap: Likewise.
commit 38f12a3085be683b598be7d10de37eba0b629f28
Author: Stefano Lattarini <address@hidden>
Date: Sun Aug 14 14:00:53 2011 +0200
testsuite: avoid confusing SKIP messages for errored tests
* tests/defs [$use_tap] (exit trap): Don't print the test plan
in case the client script exited with a non-zero status. This
avoids the display of redundant "SKIP" messages for tests with
lazy plans that error out before any test is run.
commit 0d8a113dd070db481d0070a48e6577a12d8d7149
Author: Stefano Lattarini <address@hidden>
Date: Sat Aug 13 14:29:28 2011 +0200
testsuite: keep more debugging info around in 'add-missing.tap'
* tests/add-missing.tap: Use a different directory for each test,
so that in case of failure more information is kept around.
Since we are at it, remove redundant TAP diagnostic, and define
`$am_create_testdir' to "empty" to avoid bringing in unused
auxiliary files.
commit 726540faa1a5136004eb41d5b70f61e8aa2b795f
Author: Stefano Lattarini <address@hidden>
Date: Sat Aug 13 12:39:59 2011 +0200
testsuite: more granular count of test results in our TAP library
* tests/tap-functions.sh ($tap_bad_count_): Removed, superseded
by ...
($tap_fail_count_, $tap_xpass_count_): ... these new variables,
which keep more granular counts.
($tap_pass_count_: New variable.
* tests/defs (exit trap): Adjust and simplify accordingly.
commit 836ca56434e2d63b9ad8324a0b551f789878c3f6
Author: Stefano Lattarini <address@hidden>
Date: Sat Aug 13 11:52:26 2011 +0200
testsuite: fixlets and improvements in two long TAP-based tests
* tests/depmod.tap: Clean up the subdirectories used by tests that
passed, to avoid ending up with a too big test directory. This is
especially important since, on each and every system, at least some
tests in this script are expected to be skipped (which ones exactly
depends on the system), thus causing the test directory not to be
removed when the script terminates.
* tests/instspc.tap: Likewise (with the difference that tests here
are not expected to be skipped, but to xfail). Give more explicit
error message in case of setup failure. Skip the proper number of
tests when required, to avoid spurious errors due to mismatched
plan. Also, really avoid to run the unsupported test cases when
the need to skip them is detected.
-----------------------------------------------------------------------
Summary of changes:
ChangeLog | 234 ++++++++++++++++++++
Makefile.am | 21 ++
Makefile.in | 16 ++
lib/Automake/tests/Makefile.in | 7 +-
lib/am/check.am | 11 +-
lib/tap-driver | 112 ++++++----
tests/Makefile.am | 19 ++-
tests/Makefile.in | 26 ++-
tests/add-missing.tap | 37 ++-
tests/defs | 8 +-
tests/depmod.tap | 26 ++-
tests/instspc.tap | 22 ++-
tests/parallel-tests-fork-bomb.test | 144 ++++++++++++
tests/self-check-exit.tap | 30 ++-
tests/self-check-me.tap | 44 ++--
tests/suffix10.tap | 2 +
...-together.test => tap-bailout-and-logging.test} | 32 ++-
...exit.test => tap-bailout-suppress-badexit.test} | 45 +++--
... => tap-bailout-suppress-later-diagnostic.test} | 29 +--
tests/tap-bailout-suppress-later-errors.test | 74 ++++++
tests/tap-color.test | 10 +-
tests/tap-functions.sh | 15 +-
tests/tap-global-result.test | 85 +++++++-
...ror.test => tap-missing-plan-and-bad-exit.test} | 31 +--
tests/tap-plan-corner.test | 98 +++++++-
tests/tap-plan-errors.test | 31 +++-
tests/tap-planskip-and-logging.test | 74 ++++++
...hole-badexit.test => tap-planskip-badexit.test} | 0
...-not-ok-skip.test => tap-planskip-bailout.test} | 17 +-
...purious-summary.test => tap-planskip-late.test} | 33 ++--
...ok-skip.test => tap-planskip-later-errors.test} | 19 +-
...ror.test => tap-planskip-unplanned-corner.test} | 35 ++--
tests/tap-planskip-unplanned.test | 73 ++++++
...hitespace.test => tap-planskip-whitespace.test} | 0
tests/{tap-skip-whole.test => tap-planskip.test} | 23 +--
tests/tap-skip-whole-lastline.test | 59 -----
tests/tap-todo-skip-whitespace.test | 2 +-
tests/test-driver-custom-no-html.test | 4 +-
38 files changed, 1196 insertions(+), 352 deletions(-)
create mode 100755 tests/parallel-tests-fork-bomb.test
copy tests/{tap-todo-skip-together.test => tap-bailout-and-logging.test} (61%)
copy tests/{tap-skip-whole-badexit.test => tap-bailout-suppress-badexit.test}
(56%)
copy tests/{tap-no-disable-hard-error.test =>
tap-bailout-suppress-later-diagnostic.test} (67%)
create mode 100755 tests/tap-bailout-suppress-later-errors.test
copy tests/{tap-no-disable-hard-error.test =>
tap-missing-plan-and-bad-exit.test} (65%)
create mode 100755 tests/tap-planskip-and-logging.test
rename tests/{tap-skip-whole-badexit.test => tap-planskip-badexit.test} (100%)
copy tests/{tap-not-ok-skip.test => tap-planskip-bailout.test} (75%)
copy tests/{tap-no-spurious-summary.test => tap-planskip-late.test} (59%)
copy tests/{tap-not-ok-skip.test => tap-planskip-later-errors.test} (75%)
copy tests/{tap-no-disable-hard-error.test =>
tap-planskip-unplanned-corner.test} (56%)
create mode 100755 tests/tap-planskip-unplanned.test
rename tests/{tap-skip-whole-whitespace.test => tap-planskip-whitespace.test}
(100%)
rename tests/{tap-skip-whole.test => tap-planskip.test} (89%)
delete mode 100755 tests/tap-skip-whole-lastline.test
diff --git a/ChangeLog b/ChangeLog
index d91bc59..3477d61 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,237 @@
+2011-08-14 Stefano Lattarini <address@hidden>
+
+ coverage: missing tap plan and non-zero exit status
+ * tests/tap-missing-plan-and-bad-exit.test: New test.
+ * tests/Makefile.am (tap_with_common_setup_tests): Update.
+
+2011-08-12 Stefano Lattarini <address@hidden>
+
+ testsuite: fix an unportable use of sed
+ * tests/tap-todo-skip-whitespace.test: Alternation with "\|" in
+ sed regular expressions is not portable to (at least) FreeBSD
+ sed; do not use it.
+
+2011-08-12 Stefano Lattarini <address@hidden>
+
+ testsuite: avoid spurious failure if rst2html program is missing
+ * tests/test-driver-custom-no-html.test: Account for the
+ possibility of a lacking `rst2html' program in our final
+ grepping of make output.
+
+2011-08-12 Stefano Lattarini <address@hidden>
+
+ tap: improve diagnosing and reporting of plan mismatches
+ Problem exposed by a failure in the test 'tap-plan-errors.test'.
+ * lib/tap-driver (NO_PLAN, EARLY_PLAN, LATE_PLAN): New constants.
+ (Throughout the file): Use more complex semantics for `$plan_seen',
+ which now also remember whether the plan that has been seen is an
+ "early" or "late" plan; in turn, this makes ...
+ ($tap_stopped): ... this variable redundant; remove it.
+ (handle_tap_plan): Adjust to avoid producing spurious or confusing
+ error messages about extra or mismatched "late" TAP plans.
+ * tests/tap-plan-corned.test: Adjust and extend.
+ * tests/tap-color.test: Likewise.
+
+2011-08-12 Stefano Lattarini <address@hidden>
+
+ tap: log all TAP stream, even after a "Bail out!"
+ * lib/tap-driver ($bailed_out): New global boolean variable,
+ telling whether a "Bail out!" directive has been seen or not.
+ (handle_tap_bailout): This function does not anymore stop the
+ reading from TAP stream; instead, it sets `$bailed_out' to a
+ true value, so that only the subsequent parsing of the input
+ TAP stream is stopped.
+ (finish): Remove, no more needed, its contents inlined into ...
+ (main): ... this function, with related adjustments in the code
+ flow.
+ (get_test_exit_message): Do not "flush" the input TAP stream
+ to fetch the exit status of test script, it is not anymore
+ required. Add a sanity check.
+ * tests/tap-bailout-and-logging.test: New test.
+ * tests/Makefile.am (tap_with_common_setup_tests): Update.
+
+2011-08-12 Stefano Lattarini <address@hidden>
+
+ coverage: TAP diagnostics after "Bail out!" aren't reported
+ This is compatible with the behaviour of the `prove' utility.
+ * tests/tap-bailout-suppress-later-diagnostic.test: New test.
+ * tests/Makefile.am (tap_with_common_setup_tests): Update.
+
+2011-08-12 Stefano Lattarini <address@hidden>
+
+ coverage: TAP errors after a "Bail out!" directive aren't reported
+ This is compatible with the behaviour of the `prove' utility.
+ * tests/tap-bailout-suppress-later-errors.test: New test.
+ * tests/tap-bailout-badexit.test: Renamed ...
+ * tests/tap-bailout-suppress-badexit.test: ... to this, for
+ consistency.
+ * tests/Makefile.am (tap_with_common_setup_tests): Update.
+
+2011-08-12 Stefano Lattarini <address@hidden>
+
+ testsuite: reorganize tests on TAP plans
+ * tests/tap-skip-whole.test: Rename ...
+ * tests/tap-planskip.test: ... to this
+ * tests/tap-skip-whole-badexit.test: Rename ...
+ * tests/tap-planskip-badexit.test: ... to this.
+ * tests/tap-skip-whole-bailout.test: Rename ...
+ * tests/tap-planskip-bailout.test: ... to this.
+ * tests/tap-skip-whole-bailout.test: Rename ...
+ * tests/tap-planskip-later-errors.test: ... to this.
+ * tests/tap-skip-whole-unplanned.test: Rename ...
+ * tests/tap-planskip-unplanned.test: ... to this.
+ * tests/tap-skip-whole-whitespace.test: Rename ...
+ * tests/tap-planskip-whitespace.test: ... to this.
+ * tests/tap-skip-whole-continue.test: Rename ...
+ * tests/tap-planskip-and-logging.test: ... to this, extend to try
+ more cases, and fix heading comments.
+ * tests/tap-skip-whole-lastline.test: Rename ...
+ * tests/tap-planskip-late.test: ... to this, fix heading comments,
+ and remove some checks that are just a duplication of those already
+ performed in the previous script.
+ * tests/tap-results-with-planskip.test: Rename to ...
+ * tests/tap-planskip-unplanned-corner.test: ... this, and improve
+ hading comments.
+ * tests/tap-planskip-later-errors.test: New test.
+ * tests/Makefile.am (tap_with_common_setup_tests): Update.
+
+2011-08-12 Stefano Lattarini <address@hidden>
+
+ coverage: more TAP driver consistency w.r.t. 'prove'
+ * tests/skip-results-with-planskip.test: New test, verifies that
+ test results seen in a TAP stream that has a "plan with SKIP" are
+ flagged as errors.
+ * tests/Makefile.am (tap_with_common_setup_tests): Update.
+
+2011-08-12 Stefano Lattarini <address@hidden>
+
+ tap: improve some comments in the TAP driver
+ * lib/tap-driver (handle_tap_plan): Improve comments describing
+ possible errors in a "plan with SKIP" directive.
+
+2011-08-12 Stefano Lattarini <address@hidden>
+
+ tap: non-zero exit status after "Bail out!" should not be reported
+ This new behaviour is more consistent with that of the `prove'
+ utility. Also, experience with the use of TAP in the Automake's
+ own test suite has shown that reporting non-zero exit status from
+ a script after it has issued a "Bail out!" directive is mostly
+ redundant, more introducing noise rather than conveying useful
+ information.
+ * lib/tap-driver (finish): Inline the part of it that was getting
+ the script exit status ...
+ (main): ... in here.
+ * tests/tap-bailout-badexit.test: New test.
+ * tests/Makefile.am (tap_with_common_setup_tests): Update.
+
+2011-08-11 Stefano Lattarini <address@hidden>
+
+ tap: a "plan with SKIP" after test results is an error
+ This new behaviour is both more natural and more consistent with
+ that of the `prove' utility.
+ * lib/tap-driver (handle_tap_plan): Do not stop TAP parsing when
+ a "plan with SKIP" line (e.g., "1..0 # SKIP") is encountered, and
+ do not print a "SKIP" line if some TAP result has already been
+ seen.
+ * tests/tap-skip-whole.test: Adapt.
+ * tests/tap-skip-whole-lastline.test: Likewise.
+ * tests/tap-global-result.test: Adapt and extend.
+ * tests/tap-skip-plan-errors.test: Likewise, and fix an obsolete
+ small part of a comment.
+ * tests/tap-skip-whole-bailout.test: New test.
+ * tests/tap-skip-whole-unplanned.test: Likewise.
+ * tests/tap-skip-whole-continue.test: Likewise.
+ * tests/Makefile.am (tap_with_common_setup_tests): Update.
+
+2011-08-09 Stefano Lattarini <address@hidden>
+
+ test harness: avoid possible fork bomb
+ This fixes a regression w.r.t. the master branch, exposed by
+ test 'parallel-tests-fork-bomb.test'.
+ * lib/am/check.am (am--redo-logs): Detect possible infinite
+ recursion due to a test log in $(TEST_LOGS) being the same
+ as $(TEST_SUITE_LOG).
+
+2011-08-09 Stefano Lattarini <address@hidden>
+
+ coverage: enhance test against infinite recursion in test harness
+ * tests/parallel-tests-fork-bomb.test: Enhance and extend a
+ little now, for soon-to-be-implemented more explicit and safer
+ semantics.
+
+2011-08-09 Stefano Lattarini <address@hidden>
+
+ coverage: possible infinite recursion in the test harness
+ Motivated by a regression in the 'test-protocols' branch.
+ * tests/parallel-tests-fork-bomb.test: New test, checking that
+ if $(TEST_SUITE_LOG) is in $(TEST_LOGS), we obtain a diagnosed
+ error rather than a make hang or a fork bomb.
+ * tests/Makefile.am (TESTS): Update.
+
+2011-08-09 Stefano Lattarini <address@hidden>
+
+ test defs: yet more environment cleanup
+ * tests/defs: Also unset the TEST_SUITE_LOG variable.
+
+2011-08-14 Stefano Lattarini <address@hidden>
+
+ self tests: some minor refactoring
+ * tests/self-check-exit.tap: Disable the `errexit' shell flag;
+ this allow to remove some boilerplate idioms and to make some
+ code slightly less cumbersome.
+ * tests/self-check-me.tap: Likewise.
+
+2011-08-14 Stefano Lattarini <address@hidden>
+
+ maintcheck: look for TAP-based tests without TAP plan
+ * Makefile.am (sc_tests_tap_plan): New maintainer check.
+ (syntax_check_rules): Add it.
+ * tests/suffix10.tap: Declare a TAP plan.
+ * tests/add-missing.tap: Likewise.
+
+2011-08-14 Stefano Lattarini <address@hidden>
+
+ testsuite: avoid confusing SKIP messages for errored tests
+ * tests/defs [$use_tap] (exit trap): Don't print the test plan
+ in case the client script exited with a non-zero status. This
+ avoids the display of redundant "SKIP" messages for tests with
+ lazy plans that error out before any test is run.
+
+2011-08-13 Stefano Lattarini <address@hidden>
+
+ testsuite: keep more debugging info around in 'add-missing.tap'
+ * tests/add-missing.tap: Use a different directory for each test,
+ so that in case of failure more information is kept around.
+ Since we are at it, remove redundant TAP diagnostic, and define
+ `$am_create_testdir' to "empty" to avoid bringing in unused
+ auxiliary files.
+
+2011-08-13 Stefano Lattarini <address@hidden>
+
+ testsuite: more granular count of test results in our TAP library
+ * tests/tap-functions.sh ($tap_bad_count_): Removed, superseded
+ by ...
+ ($tap_fail_count_, $tap_xpass_count_): ... these new variables,
+ which keep more granular counts.
+ ($tap_pass_count_: New variable.
+ * tests/defs (exit trap): Adjust and simplify accordingly.
+
+2011-08-13 Stefano Lattarini <address@hidden>
+
+ testsuite: fixlets and improvements in two long TAP-based tests
+ * tests/depmod.tap: Clean up the subdirectories used by tests that
+ passed, to avoid ending up with a too big test directory. This is
+ especially important since, on each and every system, at least some
+ tests in this script are expected to be skipped (which ones exactly
+ depends on the system), thus causing the test directory not to be
+ removed when the script terminates.
+ * tests/instspc.tap: Likewise (with the difference that tests here
+ are not expected to be skipped, but to xfail). Give more explicit
+ error message in case of setup failure. Skip the proper number of
+ tests when required, to avoid spurious errors due to mismatched
+ plan. Also, really avoid to run the unsupported test cases when
+ the need to skip them is detected.
+
2011-08-09 Stefano Lattarini <address@hidden>
test harness: recursive make invocations must use $(AM_MAKEFLAGS)
diff --git a/Makefile.am b/Makefile.am
index eeb3a6a..3b1751a 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -198,6 +198,7 @@ sc_tests_automake_fails \
sc_tests_plain_aclocal \
sc_tests_plain_perl \
sc_tests_required_after_defs \
+sc_tests_tap_plan \
sc_tests_overriding_macros_on_cmdline \
sc_tests_plain_sleep \
sc_tests_plain_egrep_fgrep \
@@ -537,6 +538,26 @@ sc_tests_required_after_defs:
fi; \
done
+## TAP-based test scripts should not forget to declare a TAP plan. In
+## case it is not known in advance how many tests will be run, a "lazy"
+## plan can be used; but its use should be deliberate, explicitly declared
+## with a "plan_ later" call, rather than the result of an oversight.
+## This check helps to ensure this is indeed the case.
+sc_tests_tap_plan:
+ @with_plan=`grep -l '^ *plan_ ' $(srcdir)/tests/*.tap`; \
+ with_plan=`echo $$with_plan`; \
+ ok=:; \
+ for t in $(srcdir)/tests/*.tap; do \
+ case " $$with_plan " in \
+ *" $$t "*) ;; \
+ *) ok=false; echo $$t;; \
+ esac; \
+ done; \
+ $$ok || { \
+ echo 'The tests above do not declare a TAP plan.' 1>&2; \
+ exit 1; \
+ }
+
## Overriding a Makefile macro on the command line is not portable when
## recursive targets are used. Better use an envvar. SHELL is an
## exception, POSIX says it can't come from the environment. V, DESTDIR,
diff --git a/Makefile.in b/Makefile.in
index bcebf02..d0611e4 100644
--- a/Makefile.in
+++ b/Makefile.in
@@ -327,6 +327,7 @@ sc_tests_automake_fails \
sc_tests_plain_aclocal \
sc_tests_plain_perl \
sc_tests_required_after_defs \
+sc_tests_tap_plan \
sc_tests_overriding_macros_on_cmdline \
sc_tests_plain_sleep \
sc_tests_plain_egrep_fgrep \
@@ -1272,6 +1273,21 @@ sc_tests_required_after_defs:
fi; \
done
+sc_tests_tap_plan:
+ @with_plan=`grep -l '^ *plan_ ' $(srcdir)/tests/*.tap`; \
+ with_plan=`echo $$with_plan`; \
+ ok=:; \
+ for t in $(srcdir)/tests/*.tap; do \
+ case " $$with_plan " in \
+ *" $$t "*) ;; \
+ *) ok=false; echo $$t;; \
+ esac; \
+ done; \
+ $$ok || { \
+ echo 'The tests above do not declare a TAP plan.' 1>&2; \
+ exit 1; \
+ }
+
sc_tests_overriding_macros_on_cmdline:
@if grep -E '\$$MAKE .*(SHELL=.*=|=.*SHELL=)' $(srcdir)/tests/*.test;
then \
echo 'Rewrite "$$MAKE foo=bar SHELL=$$SHELL" as "foo=bar $$MAKE -e
SHELL=$$SHELL"' 1>&2; \
diff --git a/lib/Automake/tests/Makefile.in b/lib/Automake/tests/Makefile.in
index 24bc878..558f807 100644
--- a/lib/Automake/tests/Makefile.in
+++ b/lib/Automake/tests/Makefile.in
@@ -362,7 +362,12 @@ cscope cscopelist:
am--redo-logs:
@rm -f $$redo_logs
@rm -f $$redo_results
- @$(MAKE) $(AM_MAKEFLAGS) $$redo_logs
+ @if test -n "$$am__remaking_logs"; then \
+ echo "fatal: making $(TEST_SUITE_LOG): possible infinite" \
+ "recursion detected" >&2; \
+ else \
+ am__remaking_logs=yes $(MAKE) $(AM_MAKEFLAGS) $$redo_logs; \
+ fi;
@st=0; \
errmsg="fatal: making $(TEST_SUITE_LOG): failed to create"; \
for i in $$redo_bases; do \
diff --git a/lib/am/check.am b/lib/am/check.am
index 76c5273..27388df 100644
--- a/lib/am/check.am
+++ b/lib/am/check.am
@@ -132,7 +132,16 @@ am__set_TESTS_bases = \
am--redo-logs:
@rm -f $$redo_logs
@rm -f $$redo_results
- @$(MAKE) $(AM_MAKEFLAGS) $$redo_logs
+## The use of the `am__remaking_logs' environment variable below is
+## required to ensure that we don't go into an infinite recursion in
+## case a test log in $(TEST_LOGS) is the same as $(TEST_SUITE_LOG).
+## Yes, this has already happened in practice. Sigh!
+ @if test -n "$$am__remaking_logs"; then \
+ echo "fatal: making $(TEST_SUITE_LOG): possible infinite" \
+ "recursion detected" >&2; \
+ else \
+ am__remaking_logs=yes $(MAKE) $(AM_MAKEFLAGS) $$redo_logs; \
+ fi;
## Sanity check: each unreadable or non-existent test result file should
## has been properly remade at this point, as should the corresponding log
## file.
diff --git a/lib/tap-driver b/lib/tap-driver
index 2393346..95fb78c 100755
--- a/lib/tap-driver
+++ b/lib/tap-driver
@@ -39,17 +39,22 @@ my %COLOR = (
std => "\e[m",
);
+# It's important that NO_PLAN evaluates "false" as a boolean.
+use constant NO_PLAN => 0;
+use constant EARLY_PLAN => 1;
+use constant LATE_PLAN => 2;
+
# ------------------- #
# Global variables. #
# ------------------- #
my $testno = 0; # Number of test results seen so far.
-my $plan_seen = 0; # Whether the TAP plan has been seen or not.
+my $bailed_out = 0; # Whether a "Bail out!" directive has been seen.
my $parser; # TAP parser object (will be initialized later).
-# When true, it means that the rest of the input stream cannot
-# contain any further TAP results.
-my $tap_stopped = 0;
+# Whether the TAP plan has been seen or not, and if yes, which kind
+# it is ("early" is seen before any test result, "late" otherwise).
+my $plan_seen = NO_PLAN;
# ----------------- #
# Option parsing. #
@@ -96,7 +101,6 @@ sub colored ($$);
sub copy_in_global_log ();
sub decorate_result ($);
sub extract_tap_comment ($);
-sub finish ();
sub get_global_test_result ();
sub get_test_exit_message ();
sub get_test_results ();
@@ -210,10 +214,10 @@ sub start (@)
sub get_test_exit_message ()
{
- # Flush all the remaining TAP stream, so that we can obtain the
- # exit status of the TAP producer.
- do {} while defined $parser->next;
my $wstatus = $parser->wait;
+ # Watch out for possible internal errors.
+ die "couldn't get the exit ststus of the TAP producer"
+ unless defined $wstatus;
# Return an undefined value if the producer exited with success.
return unless $wstatus;
# Otherwise, determine whether it exited with error or was terminated
@@ -233,23 +237,14 @@ sub get_test_exit_message ()
}
}
-sub finish ()
-{
- if (!$cfg{"ignore-exit"} and my $msg = get_test_exit_message)
- {
- testsuite_error $msg;
- }
- write_test_results;
- close LOG or die "closing $log_file: $!\n";
- exit 0;
-}
-
sub stringify_test_result ($)
{
my $result = shift;
my $PASS = $cfg{"expect-failure"} ? "XPASS": "PASS";
my $FAIL = $cfg{"expect-failure"} ? "XFAIL": "FAIL";
- if ($result->is_unplanned || $result->number != $testno || $tap_stopped)
+ if ($result->is_unplanned
+ || $result->number != $testno
+ || $plan_seen == LATE_PLAN)
{
return "ERROR";
}
@@ -340,7 +335,7 @@ sub handle_tap_test ($)
$string .= " $description";
}
- if ($tap_stopped)
+ if ($plan_seen == LATE_PLAN)
{
$string .= " # AFTER LATE PLAN";
}
@@ -367,28 +362,38 @@ sub handle_tap_test ($)
sub handle_tap_plan ($)
{
my $plan = shift;
- # Only one plan per stream is acceptable.
- testsuite_error "multiple test plans" if $plan_seen;
- $plan_seen = 1;
- # TAP plan must come either before or after *all* the TAP results.
- # So, if we find it after having already seen at least one TAP result,
- # set a flag signaling that no more TAP results are acceptable.
- $tap_stopped = 1 if $testno >= 1;
- # Nothing more to do, unless the plan contains a SKIP directive.
- return
- if not defined $plan->directive && length ($plan->directive) > 0;
- my $explanation = $plan->explanation ?
- "- " . $plan->explanation : undef;
- report "SKIP", $explanation;
- finish;
+ if ($plan_seen)
+ {
+ # Error, only one plan per stream is acceptable.
+ testsuite_error "multiple test plans";
+ return;
+ }
+ # The TAP plan can come before or after *all* the TAP results; we speak
+ # respectively of an "early" or a "late" plan. If we see the plan line
+ # after at least one TAP result has been seen, assume we have a late
+ # plan; in this case, any further test result seen after the plan will
+ # be flagged as an error.
+ $plan_seen = ($testno >= 1 ? LATE_PLAN : EARLY_PLAN);
+ # If $testno > 0, we have an error ("too many tests run") that will be
+ # automatically dealt with later, so don't worry about it here. If
+ # $plan_seen is true, we have an error due to a repeated plan, and that
+ # has already been dealt with above. Otherwise, we have a valid "plan
+ # with SKIP" specification, and should report it as a particular kind
+ # of SKIP result.
+ if ($plan->directive && $testno == 0)
+ {
+ my $explanation = $plan->explanation ?
+ "- " . $plan->explanation : undef;
+ report "SKIP", $explanation;
+ }
}
sub handle_tap_bailout ($)
{
my ($bailout, $msg) = ($_[0], "Bail out!");
+ $bailed_out = 1;
$msg .= " " . $bailout->explanation if $bailout->explanation;
testsuite_error $msg;
- finish;
}
sub extract_tap_comment ($)
@@ -410,6 +415,9 @@ sub main (@)
{
# Verbatim copy any input line into the log file.
print $cur->raw . "\n";
+ # Parsing of TAP input should stop after a "Bail out!" directive.
+ next if $bailed_out;
+
if ($cur->is_plan)
{
handle_tap_plan ($cur);
@@ -428,18 +436,30 @@ sub main (@)
report "#", "$comment" if length $comment;
}
}
- if (!$plan_seen)
- {
- testsuite_error "missing test plan";
- }
- elsif ($parser->tests_planned != $parser->tests_run)
+ # A "Bail out!" directive should cause us to ignore any following TAP
+ # error, as well as a non-zero exit status from the TAP producer.
+ if (!$bailed_out)
{
- my ($planned, $run) = ($parser->tests_planned, $parser->tests_run);
- my $bad_amount = $run > $planned ? "many" : "few";
- testsuite_error (sprintf "too %s tests run (expected %d, got %d)",
- $bad_amount, $planned, $run);
+ if (!$plan_seen)
+ {
+ testsuite_error "missing test plan";
+ }
+ elsif ($parser->tests_planned != $parser->tests_run)
+ {
+ my ($planned, $run) = ($parser->tests_planned, $parser->tests_run);
+ my $bad_amount = $run > $planned ? "many" : "few";
+ testsuite_error (sprintf "too %s tests run (expected %d, got %d)",
+ $bad_amount, $planned, $run);
+ }
}
- finish;
+ if (!$cfg{"ignore-exit"} && !$bailed_out)
+ {
+ my $msg = get_test_exit_message ();
+ testsuite_error $msg if $msg;
+ }
+ write_test_results;
+ close LOG or die "closing $log_file: $!\n";
+ exit 0;
}
# ----------- #
diff --git a/tests/Makefile.am b/tests/Makefile.am
index f2e9c2d..0db204e 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -742,6 +742,7 @@ parallel-tests-log-override-2.test \
parallel-tests-log-override-recheck.test \
parallel-tests-log-compiler-example.test \
parallel-tests-cmdline-override.test \
+parallel-tests-fork-bomb.test \
parallel-tests-empty-testlogs.test \
parallel-tests-driver-install.test \
parallel-tests-no-color-in-log.test \
@@ -1153,6 +1154,10 @@ testsuite-summary-count-many.log:
extract-testsuite-summary
tap_with_common_setup_tests = \
tap-autonumber.test \
tap-bailout.test \
+tap-bailout-and-logging.test \
+tap-bailout-suppress-badexit.test \
+tap-bailout-suppress-later-diagnostic.test \
+tap-bailout-suppress-later-errors.test \
tap-color.test \
tap-deps.test \
tap-diagnostic.test \
@@ -1182,12 +1187,18 @@ tap-plan.test \
tap-plan-corner.test \
tap-plan-errors.test \
tap-plan-middle.test \
+tap-missing-plan-and-bad-exit.test \
+tap-planskip.test \
+tap-planskip-late.test \
+tap-planskip-and-logging.test \
+tap-planskip-unplanned.test \
+tap-planskip-unplanned-corner.test \
+tap-planskip-whitespace.test \
+tap-planskip-badexit.test \
+tap-planskip-bailout.test \
+tap-planskip-later-errors.test \
tap-realtime.test \
tap-recheck-logs.test \
-tap-skip-whole-whitespace.test \
-tap-skip-whole.test \
-tap-skip-whole-lastline.test \
-tap-skip-whole-badexit.test \
tap-todo-skip-together.test \
tap-todo-skip-whitespace.test \
tap-todo-skip.test \
diff --git a/tests/Makefile.in b/tests/Makefile.in
index 2821858..5f50e7d 100644
--- a/tests/Makefile.in
+++ b/tests/Makefile.in
@@ -985,6 +985,7 @@ parallel-tests-log-override-2.test \
parallel-tests-log-override-recheck.test \
parallel-tests-log-compiler-example.test \
parallel-tests-cmdline-override.test \
+parallel-tests-fork-bomb.test \
parallel-tests-empty-testlogs.test \
parallel-tests-driver-install.test \
parallel-tests-no-color-in-log.test \
@@ -1380,6 +1381,10 @@ yflags-var-expand.test
tap_with_common_setup_tests = \
tap-autonumber.test \
tap-bailout.test \
+tap-bailout-and-logging.test \
+tap-bailout-suppress-badexit.test \
+tap-bailout-suppress-later-diagnostic.test \
+tap-bailout-suppress-later-errors.test \
tap-color.test \
tap-deps.test \
tap-diagnostic.test \
@@ -1409,12 +1414,18 @@ tap-plan.test \
tap-plan-corner.test \
tap-plan-errors.test \
tap-plan-middle.test \
+tap-missing-plan-and-bad-exit.test \
+tap-planskip.test \
+tap-planskip-late.test \
+tap-planskip-and-logging.test \
+tap-planskip-unplanned.test \
+tap-planskip-unplanned-corner.test \
+tap-planskip-whitespace.test \
+tap-planskip-badexit.test \
+tap-planskip-bailout.test \
+tap-planskip-later-errors.test \
tap-realtime.test \
tap-recheck-logs.test \
-tap-skip-whole-whitespace.test \
-tap-skip-whole.test \
-tap-skip-whole-lastline.test \
-tap-skip-whole-badexit.test \
tap-todo-skip-together.test \
tap-todo-skip-whitespace.test \
tap-todo-skip.test \
@@ -1509,7 +1520,12 @@ cscope cscopelist:
am--redo-logs:
@rm -f $$redo_logs
@rm -f $$redo_results
- @$(MAKE) $(AM_MAKEFLAGS) $$redo_logs
+ @if test -n "$$am__remaking_logs"; then \
+ echo "fatal: making $(TEST_SUITE_LOG): possible infinite" \
+ "recursion detected" >&2; \
+ else \
+ am__remaking_logs=yes $(MAKE) $(AM_MAKEFLAGS) $$redo_logs; \
+ fi;
@st=0; \
errmsg="fatal: making $(TEST_SUITE_LOG): failed to create"; \
for i in $$redo_bases; do \
diff --git a/tests/add-missing.tap b/tests/add-missing.tap
index 4c6a624..36f7676 100755
--- a/tests/add-missing.tap
+++ b/tests/add-missing.tap
@@ -19,9 +19,13 @@
# the files) it's supposed to, and that these files are symlinked by
# default, but copied if the `--copy' option is used.
+am_create_testdir=empty
. ./defs || Exit 1
+plan_ "later"
+
build_aux=build-aux
+ocwd=`pwd` || fatal_ "cannot get current working directory"
# Try to improve readability of displayed diffs.
if diff -u /dev/null /dev/null; then
@@ -40,7 +44,7 @@ AC_CONFIG_FILES([Makefile])
END
# Pre-compute aclocal.m4, in order to save several aclocal invocations.
-cat >> configure.in <<'END'
+cat configure.stub - > configure.in <<'END'
AC_PROG_CC
AC_CANONICAL_BUILD
AC_CANONICAL_HOST
@@ -51,17 +55,22 @@ AM_PATH_PYTHON
END
$ACLOCAL || framework_failure_ "cannot pre-compute aclocal.m4"
-rm -rf install-sh missing depcomp configure.in autom4te*.cache
+rm -rf configure.in autom4te*.cache
mv aclocal.m4 aclocal.stub
+# For debugging.
cat configure.stub
cat aclocal.stub
+check_count=0
+
# This is hacky and ugly and complex, but allow us to organize our tests
# below in a more "declarative fashion". All in all, a good trade-off.
check_ ()
{
set +x # Temporary disable shell traces to remove noise from log files.
+ incr_ check_count
+ echo check count: $check_count
override=no
run_aclocal=no
fetch_file=
@@ -74,17 +83,21 @@ check_ ()
esac
shift
done
- mkdir testdir-generic
- cd testdir-generic
+ mkdir testdir-$check_count
+ cd testdir-$check_count
+ # Directory for common data files (specific to the current test, but
+ # shared by its "subtests").
+ mkdir generic
+ cd generic
# Use `echo > ...', not `: > ...', here and below, as Solaris 10 /bin/sh
# might not execute the latter (the Autoconf manual gives more details).
echo > Makefile.am
if test $override = yes; then
echo > configure.in
else
- cp ../configure.stub configure.in
+ cp "$ocwd"/configure.stub configure.in
fi
- test -z "$fetch_file" || cp ../"$fetch_file" .
+ test -z "$fetch_file" || cp "$ocwd/$fetch_file" .
# Read description of "test scenario" from standard input.
what= line= name= files=
while read line; do
@@ -107,7 +120,6 @@ check_ ()
esac
done
test -n "$name" || fatal_ "name of a test case not specified"
- diag_ "Will run checks for $name."
if test $run_aclocal = yes; then
if $ACLOCAL; then
ok_ "[$name] aclocal.m4 rebuilt"
@@ -117,7 +129,7 @@ check_ ()
not_ok_ "[$name] aclocal failure, aclocal.m4 not rebuilt"
fi
else
- cp ../aclocal.stub aclocal.m4
+ cp "$ocwd"/aclocal.stub aclocal.m4
fi
echo == Makefile.am ==
cat Makefile.am
@@ -128,7 +140,8 @@ check_ ()
cd ..
# End of "test scenario" setup.
set -x # Re-enable shell traces.
- ls -l testdir-generic
+ pwd
+ ls -l generic
# Test once with `--copy', once without.
for action in link copy; do
case $action in
@@ -137,8 +150,8 @@ check_ ()
*) fatal_ "invalid value '$action' for \$action";;
esac
pfx="[$action $name]"
- cp -R testdir-generic testdir-$action
- cd testdir-$action
+ cp -R generic $action
+ cd $action
# If the required auxiliary files are missing, and automake is
# not told to install them, it should complain and error out,
# and also give a useful suggestion.
@@ -202,7 +215,7 @@ check_ ()
AUTOMAKE_run -d "$pfx automake finds all added files"
cd ..
done # for action in link copy
- rm -rf testdir-*
+ cd "$ocwd" || fatal_ "cannot chdir back to top-level test directory"
}
# Automake should always and unconditionally require the `missing'
diff --git a/tests/defs b/tests/defs
index 047e22f..ac9797c 100644
--- a/tests/defs
+++ b/tests/defs
@@ -154,6 +154,7 @@ unset AM_COLOR_TESTS
unset TESTS
unset TEST_LOG_COMPILER
unset TEST_LOGS
+unset TEST_SUITE_LOG
unset RECHECK_LOGS
unset VERBOSE
@@ -868,13 +869,10 @@ if test "$sh_errexit_works" = yes; then
keep_testdirs=no
fi
if test $using_tap = yes; then
- if test $have_tap_plan_ != yes; then
+ if test $have_tap_plan_ != yes && test $exit_status -eq 0; then
plan_ "now"
fi
- test $exit_status -eq 0 \
- && test $tap_xfail_count_ -eq 0 \
- && test $tap_skip_count_ -eq 0 \
- && test $tap_bad_count_ -eq 0 \
+ test $exit_status -eq 0 && test $tap_pass_count_ -eq $tap_count_ \
|| keep_testdirs=yes
else
# This is to ensure that a test script does give a SKIP outcome just
diff --git a/tests/depmod.tap b/tests/depmod.tap
index 82fd844..c40fea8 100755
--- a/tests/depmod.tap
+++ b/tests/depmod.tap
@@ -146,28 +146,44 @@ for vpath in simple long absolute; do
mkdir $depmode-$vpath.d
cd $depmode-$vpath.d
- command_ok_ "$pfx configure" \
- "$srcdir"/configure am_cv_CC_dependencies_compiler_type=$depmode
+ if "$srcdir"/configure am_cv_CC_dependencies_compiler_type=$depmode
+ then
+ r0='ok'
+ else
+ r0='not ok'
+ fi
+ result_ "$r0" "$pfx configure" \
# Do not fail with the first call to make fails, as the depmode
# we've forced might not actually work, but we have overridden the
# _AM_DEPENDENCIES tests.
+ r1='not ok'
if $MAKE; then
# We must clean and rebuild, as the actual error only happens the
# second time the objects are built because 'depcomp' has silently
# messed up the .Po files the first time.
- r=ok
$MAKE clean \
&& { e=0; $MAKE >output 2>&1 || e=1; cat output; test $e -eq 0; } \
&& { if grep 'src/[._]deps' output; then false; else :; fi; } \
- || r='not ok'
- result_ "$r" "$pfx make & remake"
+ && r1='ok'
+ result_ "$r1" "$pfx make & remake"
else
skip_ -r "can't force depmode" "$pfx make & remake"
fi
cd "$ocwd" || fatal_ "cannot chdir back to top directory"
+ # Remove subdirectories for tests that have passed, to avoid ending up
+ # with a too big test directory. This is especially important since,
+ # on each and every system, at least some tests in this script are
+ # expected to be skipped (which ones exactly depends on the system),
+ # thus causing the test directory not to be removed when the script
+ # terminates.
+ case $keep_testdirs,$r0,$r1 in
+ ,ok,ok) rm_rf_ $depmode-$vpath.d;;
+ *) : For lesser shells with broken 'set -e';;
+ esac
+
done
done
diff --git a/tests/instspc.tap b/tests/instspc.tap
index dc52299..46f4b4e 100755
--- a/tests/instspc.tap
+++ b/tests/instspc.tap
@@ -136,9 +136,9 @@ EOF
test -f '$(DESTDIR)/$(file)-prefix/foo/libbase.a'
EOF
- $ACLOCAL
- $AUTOCONF
- $AUTOMAKE -a
+ $ACLOCAL || framework_failure_ "aclocal failed"
+ $AUTOCONF || framework_failure_ "autoconf failed"
+ $AUTOMAKE || framework_failure_ "automake failed"
}
# ================= #
@@ -223,8 +223,11 @@ for test_name in $test_names_list; do
# Skip the next checks if this system doesn't support the required
# characters in file names.
- mkdir "./$test_string" || \
- skip_ -r "mkdir failed" "$test_name"
+ mkdir "./$test_string" || {
+ skip_ -r "mkdir failed" "$test_name in builddir"
+ skip_ -r "mkdir failed" "$test_name in destdir"
+ continue
+ }
# Where are the "weird" characters going to be used, in $(builddir)
# or in $(DESTDIR)? They are always going to be used in $(prefix)
@@ -271,6 +274,15 @@ for test_name in $test_names_list; do
cd "$ocwd" || fatal_ "cannot chdir back to test directory"
+ # Remove subdirectories for tests that have passed, to avoid ending up
+ # with a too big test directory. This is especially important since
+ # some tests in this tests are expected to fail, and this will cause
+ # the test directory not to be removed when the script terminates.
+ case $keep_testdirs,$r1 in
+ ,ok) rm_rf_ $depmode-$vpath.d;;
+ *) : For lesser shells with broken 'set -e';;
+ esac
+
done # $instspc_action
done # $test_name
diff --git a/tests/parallel-tests-fork-bomb.test
b/tests/parallel-tests-fork-bomb.test
new file mode 100755
index 0000000..7b616c1
--- /dev/null
+++ b/tests/parallel-tests-fork-bomb.test
@@ -0,0 +1,144 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Check parallel-tests features:
+# - If $(TEST_SUITE_LOG) is in $(TEST_LOGS), we get a diagnosed
+# error, not a make hang or a system freeze.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+# The tricky part of this test is to avoid that make hangs or even
+# freezes the system in case infinite recursion (which is the bug we
+# are testing against) is encountered. The following hacky makefile
+# should minimize the probability of that happening.
+cat > Makefile.am << 'END'
+TEST_LOG_COMPILER = true
+TESTS =
+
+errmsg = ::OOPS:: Recursion too deep
+
+if IS_GNU_MAKE
+
+ is_too_deep := $(shell test $(MAKELEVEL) -lt 10 && echo no)
+
+## Indenteation here required to avoid confusing Automake.
+ ifeq ($(is_too_deep),no)
+ else
+ $(error $(errmsg), $(MAKELEVEL) levels)
+ endif
+
+else !IS_GNU_MAKE
+
+# We use mkdir to detect the level of recursion, since it is easy
+# to use and assured to be portably atomical. Also use an higher
+# number than with GNU make above, since the level used here can
+# be incremented by tow or more per recursion.
+recursion-not-too-deep:
+ @ok=no; \
+ for i in 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 \
+ 18 19 20 21 22 23 24 25 26 27 28 29; \
+ do \
+ echo " mkdir rec-$$i.d"; \
+ if mkdir rec-$$i.d; then \
+ ok=yes; break; \
+ else :; fi; \
+ done; \
+ test $$ok = yes || { echo '$(errmsg)' >&2; exit 1; }
+.PHONY: recursion-not-too-deep
+clean-local:
+ rmdir rec-[0-9].d
+
+targets = all check recheck $(TESTS) $(TEST_LOGS) $(TEST_SUITE_LOG)
+$(targets): recursion-not-too-deep
+
+# For BSD make.
+.BEGIN: recursion-not-too-deep
+
+endif !IS_GNU_MAKE
+END
+
+if using_gmake; then
+ cond=:
+else
+ cond=false
+fi
+
+cat >> configure.in << END
+AM_CONDITIONAL([IS_GNU_MAKE], [$cond])
+AC_OUTPUT
+END
+
+# Another helpful idiom to avoid hanging on capable systems. The subshell
+# is needed since `ulimit' might be a special shell builtin.
+if (ulimit -t 8); then ulimit -t 8; fi
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE -a -Wno-portability
+
+./configure
+
+do_check ()
+{
+ st=0
+ log=$1; shift
+ env "$@" $MAKE -e check >output 2>&1 || st=$?
+ cat output
+ $FGREP '::OOPS::' output && Exit 1 # Possible infinite recursion.
+ # Check that at least we don't create a botched global log file.
+ test ! -f "$log"
+ if using_gmake; then
+ grep "[Cc]ircular.*dependency" output | $FGREP "$log"
+ test $st -gt 0
+ else
+ # Look for possible error messages about circular dependencies from
+ # either make or our own recipes. At least one such a message must
+ # be present. OTOH, some make implementations (e.g., NetBSD's), while
+ # smartly detecting the circular dependency early and diagnosing it,
+ # still exit with a successful exit status (yikes!). So don't check
+ # the exit status of non-GNU make, to avoid spurious failures.
+ # this case.
+ err_seen=no
+ for err_rx in \
+ 'circular.* depend' \
+ 'depend.* circular' \
+ 'graph cycle' \
+ 'infinite (loop|recursion)' \
+ 'depend.* on itself' \
+ ; do
+ $EGREP -i "$err_rx" output | $FGREP "$log" || continue
+ err_seen=yes
+ break
+ done
+ test $err_seen = yes || Exit 1
+ fi
+}
+
+: > test-suite.test
+do_check test-suite.log TESTS=test-suite.test
+rm -f *.log *.test
+
+: > 0.test
+: > 1.test
+: > 2.test
+: > 3.test
+: > foobar.test
+do_check foobar.log TEST_LOGS='0.log 1.log foobar.log 2.log 3.log' \
+ TEST_SUITE_LOG=foobar.log
+rm -f *.log *.test
+
+:
diff --git a/tests/self-check-exit.tap b/tests/self-check-exit.tap
index fc3fdca..6f35e08 100755
--- a/tests/self-check-exit.tap
+++ b/tests/self-check-exit.tap
@@ -24,14 +24,19 @@ am_create_testdir=no
plan_ 34
+# This test becomes more cumbersome if we keep the `errexit' shell flag
+# set. And removing it is no big deal, as this test is a TAP-based one,
+# so that false positives remain very unlikely.
+set +e
+
AM_TESTS_REEXEC=no; export AM_TESTS_REEXEC
init='. ./defs; stderr_fileno_=2; unset am_explicit_skips'
for st in 1 2 3 4 5 77 99 126 127 128 129 130 255; do
for exit_cmd in "Exit $st" "sh -c 'exit $st'"; do
- rc=0; $SHELL -c "$init; $exit_cmd; :" || rc=$?
- command_ok_ "$exit_cmd" test $rc -eq $st
+ $SHELL -c "$init; $exit_cmd; :"
+ command_ok_ "$exit_cmd" test $? -eq $st
done
done
@@ -41,7 +46,8 @@ for sig in 1 2 13 15; do
# signal when one is sent to the child $SHELL.
trap : 2
fi
- rc=0; $SHELL -c "$init; kill -$sig \$\$; :" || rc=$?
+ $SHELL -c "$init; kill -$sig \$\$; :"
+ rc=$?
if test $sig -eq 2; then
# Reset default SIGINT handler as portably as possible.
trap 2 || trap - 2
@@ -58,7 +64,7 @@ for sig in 1 2 13 15; do
# signal shall be reported as greater than 128".
command_ok_ "kill -$sig" test $rc -gt 128
fi
-
+ unset rc
done
: Non-existent program.
@@ -74,20 +80,20 @@ if $SHELL -c 'set -e; trap "exit \$?" 0;
non-existent-program'; then
else
maybe_todo="" reason=""
fi
-rc=0; $SHELL -c "$init; non-existent-program; :" || rc=$?
+$SHELL -c "$init; non-existent-program; :"
command_ok_ "command not found" -D "$maybe_todo" -r "$reason" \
- -- test $rc -gt 0
+ -- test $? -gt 0
: Non-executable command.
test -f Makefile && test ! -x Makefile || \
framowork_failure_ "no proper Makefile in the current directory"
-rc=0; $SHELL -c "$init; ./Makefile; :" || rc=$?
-command_ok_ "permission denied" test $rc -gt 0
+$SHELL -c "$init; ./Makefile; :"
+command_ok_ "permission denied" test $? -gt 0
: Syntax errors in the test code.
-rc=0; $SHELL -c "$init; if :; then" || rc=$?
-command_ok_ "syntax error 1" test $rc -gt 0
-rc=0; $SHELL -c "$init; fi" || rc=$?
-command_ok_ "syntax error 2" test $rc -gt 0
+$SHELL -c "$init; if :; then"
+command_ok_ "syntax error 1" test $? -gt 0
+$SHELL -c "$init; fi"
+command_ok_ "syntax error 2" test $? -gt 0
:
diff --git a/tests/self-check-me.tap b/tests/self-check-me.tap
index 503a1d0..d7f61d6 100755
--- a/tests/self-check-me.tap
+++ b/tests/self-check-me.tap
@@ -24,17 +24,16 @@ am_create_testdir=no
plan_ 9
AM_TESTS_REEXEC=no; export AM_TESTS_REEXEC
+
+# This test becomes cumbersome if we keep the `errexit' shell flag set.
+# And removing it is no big deal, as this test is a TAP-based one, so
+# that false positives remain very unlikely.
set +e
do_check ()
{
- if $SHELL -c '. ./defs && echo me=$me' "$1" | grep "^me=$2$"; then
- r='ok'
- else
- r='not ok'
- fi
- result_ "$r" "me=$1"
- unset r
+ $SHELL -c '. ./defs && echo me=$me' "$1" | grep "^me=$2$"
+ command_ok_ "me=$1" test $? -eq 0
}
do_check foo-bar-.test 'foo-bar-'
@@ -45,31 +44,26 @@ do_check a.b.c.test 'a\.b\.c'
# If we override $me, ./defs should not modify it.
-if s=`$SHELL -c 'me=foo.test && . ./defs && echo me=$me' bad.test`; then
- r='ok'
-else
- r='not ok'
-fi
-result_ "$r" "override of \$me before ./defs causes no error"
+s=`$SHELL -c 'me=foo.test && . ./defs && echo me=$me' bad.test`
+command_ok_ "override of \$me before ./defs causes no error" \
+ test $? -eq 0
r='ok'
-if printf '%s\n' "$s" | grep '^me=foo\.test$'; then :; else r='not ok'; fi
-if printf '%s\n' "$s" | grep 'me=bad'; then r='not ok'; else :; fi
+printf '%s\n' "$s" | grep '^me=foo\.test$' || r='not ok'
+printf '%s\n' "$s" | grep 'me=bad' && r='not ok'
result_ "$r" "override of \$me before defs is honored"
+unset r
# Overriding $me after sourcing ./defs-static should work.
-
-if s=`$SHELL -c '. ./defs-static && me=zardoz &&
- . ./defs && echo me=$me' bad.test`; then
- r='ok'
-else
- r='not ok'
-fi
-result_ "$r" "override of \$me after defs-static causes no error"
+s=`$SHELL -c '. ./defs-static && me=zardoz &&
+ . ./defs && echo me=$me' bad.test`
+command_ok_ "override of \$me after ./defs-static causes no error" \
+ test $? -eq 0
r='ok'
-if printf '%s\n' "$s" | grep '^me=zardoz$'; then :; else r='not ok'; fi
-if printf '%s\n' "$s" | grep 'me=bad'; then r='not ok'; else :; fi
+printf '%s\n' "$s" | grep '^me=zardoz$' || r='not ok'
+printf '%s\n' "$s" | grep 'me=bad' && r='not ok'
result_ "$r" "override of \$me after ./defs-static is honored"
+unset r
:
diff --git a/tests/suffix10.tap b/tests/suffix10.tap
index d71450d..2999524 100755
--- a/tests/suffix10.tap
+++ b/tests/suffix10.tap
@@ -20,6 +20,8 @@
required='cc libtoolize yacc'
. ./defs || Exit 1
+plan_ 7
+
cat >>configure.in <<EOF
AC_PROG_CC
AC_PROG_YACC
diff --git a/tests/tap-todo-skip-together.test
b/tests/tap-bailout-and-logging.test
similarity index 61%
copy from tests/tap-todo-skip-together.test
copy to tests/tap-bailout-and-logging.test
index 14b8ed5..9854ea8 100755
--- a/tests/tap-todo-skip-together.test
+++ b/tests/tap-bailout-and-logging.test
@@ -15,7 +15,8 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
# TAP support:
-# - TODO and SKIP directives on the same line: the first one wins
+# - even after a "Bail out!" directive, all input is still copied in
+# the log file
parallel_tests=yes
. ./defs || Exit 1
@@ -23,17 +24,26 @@ parallel_tests=yes
. "$testsrcdir"/tap-setup.sh || fatal_ "sourcing tap-setup.sh"
cat > all.test <<END
-1..2
-ok 1 # SKIP TODO
-not ok 2 # TODO SKIP
+First line
+Bail out!
+non-TAP line after bailout
+# TAP diagnostic after bailout
+1..0 # SKIP TAP plan after bailout
+ok 1 - TAP result after bailout
END
-$MAKE check >stdout || { cat stdout; Exit 1; }
-cat stdout
-
-count_test_results total=2 pass=0 fail=0 xpass=0 xfail=1 skip=1 error=0
-
-grep '^SKIP: all\.test 1 ' stdout
-grep '^XFAIL: all\.test 2 ' stdout
+$MAKE check && { cat all.log; Exit 1; }
+cat all.log
+
+for rx in \
+ 'First line' \
+ 'Bail out!' \
+ 'non-TAP line after bailout' \
+ '# TAP diagnostic after bailout' \
+ '1\.\.0 # SKIP TAP plan after bailout' \
+ 'ok 1 - TAP result after bailout' \
+; do
+ grep "^$rx$" all.log
+done
:
diff --git a/tests/tap-skip-whole-badexit.test
b/tests/tap-bailout-suppress-badexit.test
similarity index 56%
copy from tests/tap-skip-whole-badexit.test
copy to tests/tap-bailout-suppress-badexit.test
index 76ba692..059394c 100755
--- a/tests/tap-skip-whole-badexit.test
+++ b/tests/tap-bailout-suppress-badexit.test
@@ -14,39 +14,52 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
-# TAP support:
-# - an exit status != 0 of a test script causes an hard error, even if
-# the last line of output is a "SKIP plan" (e.g., "1..0 # SKIP").
+# Basic TAP test protocol support:
+# - A "Bail out!" directive causes the driver to ignore the exit
+# status of the test script.
parallel_tests=yes
. ./defs || Exit 1
-echo TESTS = one.test two.test > Makefile.am
+tests='exit.test exit127.test sighup.test sigterm.test'
+
+cat > Makefile.am <<END
+TESTS = $tests
+END
. "$testsrcdir"/tap-setup.sh || fatal_ "sourcing tap-setup.sh"
-cat > one.test <<'END'
+cat > exit.test << 'END'
#!/bin/sh
-echo '1..0 # SKIP'
+echo 'Bail out!'
exit 1
END
-cat > two.test <<'END'
+cat > exit127.test << 'END'
#!/bin/sh
-echo '1..0'
-exit 22
+echo 'Bail out!'
+exit 127
END
-chmod a+x one.test two.test
+cat > sighup.test << 'END'
+#!/bin/sh
+echo 'Bail out!'
+kill -1 $$
+END
+
+cat > sigterm.test << 'END'
+#!/bin/sh
+echo 'Bail out!'
+kill -15 $$
+END
+
+chmod a+x $tests
$MAKE check >stdout && { cat stdout; Exit 1; }
cat stdout
-# The 'prove' utility reports both the skip and the non-zero exit status,
-# so we do the same.
-count_test_results total=4 pass=0 fail=0 xpass=0 xfail=0 skip=2 error=2
-
-grep '^ERROR: one\.test - exited with status 1$' stdout
-grep '^ERROR: two\.test - exited with status 22$' stdout
+count_test_results total=4 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=4
+for tst in $tests; do grep "^ERROR: $tst - Bail out!" stdout; done
+$EGREP "ERROR: .*(exit.*status|terminat.*signal)" stdout && Exit 1
:
diff --git a/tests/tap-no-disable-hard-error.test
b/tests/tap-bailout-suppress-later-diagnostic.test
similarity index 67%
copy from tests/tap-no-disable-hard-error.test
copy to tests/tap-bailout-suppress-later-diagnostic.test
index 678e184..679343f 100755
--- a/tests/tap-no-disable-hard-error.test
+++ b/tests/tap-bailout-suppress-later-diagnostic.test
@@ -15,36 +15,31 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
# TAP support:
-# - "Bail out!" magic and TAP parse errors are not disabled nor turned
-# into simple failures by the definition DISABLE_HARD_ERRORS.
+# - A "Bail out!" directive causes the driver to ignore any TAP
+# diagnostic message in the rest of the following TAP stream.
parallel_tests=yes
. ./defs || Exit 1
-cat > Makefile.am << 'END'
-DISABLE_HARD_ERRORS = yes
-TEST_LOG_COMPILER = cat
-TESTS = bail.test few.test noplan.test
-END
-
. "$testsrcdir"/tap-setup.sh || fatal_ "sourcing tap-setup.sh"
-cat > bail.test <<END
-1..1
-Bail out!
-END
+echo AM_TEST_LOG_DRIVER_FLAGS = --comments >> Makefile
-cat > few.test <<END
+cat > all.test <<END
1..1
+# sanity check
+ok 1
+Bail out!
+# not seen
END
-cat > noplan.test <<END
-# nothing here
-END
+# Doing the sums above, we have:
$MAKE check >stdout && { cat stdout; Exit 1; }
cat stdout
-count_test_results total=3 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=3
+count_test_results total=2 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=1
+grep '# all.test: sanity check' stdout
+grep 'not seen' stdout && Exit 1
:
diff --git a/tests/tap-bailout-suppress-later-errors.test
b/tests/tap-bailout-suppress-later-errors.test
new file mode 100755
index 0000000..19aa9e2
--- /dev/null
+++ b/tests/tap-bailout-suppress-later-errors.test
@@ -0,0 +1,74 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - A "Bail out!" directive causes the driver to ignore any TAP
+# result or error in the rest of the following TAP stream.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+. "$testsrcdir"/tap-setup.sh || fatal_ "sourcing tap-setup.sh"
+
+# Various errors that can all be squashed into a single test script.
+cat > foo.test << 'END'
+1..5
+Bail out!
+# All possible test results.
+# Test out-of-order.
+ok 4
+# Extra TAP plan.
+1..2
+# Another bailout directive.
+Bail out! Not seen.
+# Stop now, with too few tests run.
+END
+
+# Tests run after a "SKIP" plan.
+cat > bar.test << 'END'
+1..0 # SKIP
+Bail out!
+ok 1
+END
+
+# Too many tests run.
+cat > baz.test << 'END'
+1..1
+ok 1
+Bail out!
+ok 2
+ok 3
+END
+
+TESTS='foo.test bar.test baz.test' $MAKE -e check >stdout \
+ && { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=5 pass=1 fail=0 xpass=0 xfail=0 skip=1 error=3
+
+grep '^ERROR: foo\.test - Bail out!$' stdout
+grep '^ERROR: bar\.test - Bail out!$' stdout
+grep '^SKIP: bar\.test' stdout
+grep '^ERROR: baz\.test - Bail out!$' stdout
+grep '^PASS: baz\.test 1$' stdout
+
+$FGREP 'Not seen' stdout && Exit 1
+
+test `$FGREP -c ': foo.test' stdout` -eq 1
+test `$FGREP -c ': bar.test' stdout` -eq 2
+test `$FGREP -c ': baz.test' stdout` -eq 2
+
+:
diff --git a/tests/tap-color.test b/tests/tap-color.test
index cd0329b..16fea15 100755
--- a/tests/tap-color.test
+++ b/tests/tap-color.test
@@ -44,7 +44,7 @@ AUTOMAKE_OPTIONS = color-tests
AM_TEST_LOG_DRIVER_FLAGS = --comments
TEST_LOG_COMPILER = cat
TESTS = all.test skip.test bail.test badplan.test noplan.test \
- few.test many.test order.test
+ few.test many.test order.test afterlate.test
END
. "$testsrcdir"/tap-setup.sh || fatal_ "sourcing tap-setup.sh"
@@ -96,6 +96,12 @@ cat > order.test << 'END'
ok 5
END
+cat > afterlate.test << 'END'
+ok 1
+1..2
+ok 2
+END
+
AM_COLOR_TESTS=always $MAKE check >stdout && { cat stdout; Exit 1; }
cat stdout
@@ -111,12 +117,12 @@ test_color ()
cat stdout | grep "^${grn}PASS${std}: bail\.test 1$"
cat stdout | grep "^${mgn}ERROR${std}: bail\.test - Bail out!$"
cat stdout | grep "^${mgn}ERROR${std}: badplan\.test - multiple test plans$"
- cat stdout | grep "^${mgn}ERROR${std}: badplan\.test 2 # AFTER LATE PLAN$"
cat stdout | grep "^${mgn}ERROR${std}: noplan\.test - missing test plan$"
cat stdout | grep "^${mgn}ERROR${std}: few.test - too few tests run
(expected 2, got 1)$"
cat stdout | grep "^${mgn}ERROR${std}: many.test - too many tests run
(expected 1, got 2)$"
cat stdout | grep "^${mgn}ERROR${std}: many.test 2 # UNPLANNED$"
cat stdout | grep "^${mgn}ERROR${std}: order.test 5 # OUT-OF-ORDER
(expecting 1)$"
+ cat stdout | grep "^${mgn}ERROR${std}: afterlate\.test 2 # AFTER LATE PLAN$"
# Diagnostic messages shouldn't be colorized.
cat stdout | grep "^# all\.test: Hi! I shouldn't be colorized!$"
:
diff --git a/tests/tap-functions.sh b/tests/tap-functions.sh
index 603b53c..ea88887 100644
--- a/tests/tap-functions.sh
+++ b/tests/tap-functions.sh
@@ -22,14 +22,14 @@
# from within a subshell, unless explicitly noted otherwise.
#
-# The count of the TAP test results seen so far.
+# The counts of the TAP test results seen so far: total count and
+# per-result counts.
tap_count_=0
-# The count of skipped tests.
+tap_pass_count_=0
tap_skip_count_=0
-# The count of tests that experienced an expected failure.
+tap_fail_count_=0
tap_xfail_count_=0
-# The count of tests with unexpected outcomes (i.e., failed and xpassed).
-tap_bad_count_=0
+tap_xpass_count_=0
# The first "test -n" tries to avoid extra forks when possible.
if test -n "${ZSH_VERSION}${BASH_VERSION}" \
@@ -132,9 +132,10 @@ result_ ()
esac
incr_ tap_count_
case $tap_result_,$tap_directive_ in
- ok,) ;; # Passed.
+ ok,) incr_ tap_pass_count_;; # Passed.
not\ ok,TODO) incr_ tap_xfail_count_;; # Expected failure.
- not\ ok,*|ok,TODO) incr_ tap_bad_count_ ;; # Failed or xpassed.
+ not\ ok,*) incr_ tap_fail_count_ ;; # Failed.
+ ok,TODO) incr_ tap_xpass_count_ ;; # Unexpected pass.
ok,SKIP) incr_ tap_skip_count_ ;; # Skipped.
*) bailout_ "internal error in 'result_'";; # Can't happen.
esac
diff --git a/tests/tap-global-result.test b/tests/tap-global-result.test
index 32e2401..7b83e6a 100755
--- a/tests/tap-global-result.test
+++ b/tests/tap-global-result.test
@@ -39,7 +39,8 @@ END
cat > skipall.test <<'END'
1..0 # SKIP
-not ok 1
+foo
+# bar
END
cat > fail.test <<'END'
@@ -56,8 +57,70 @@ END
(sed '1s/.*/1..4/' ok.test && echo 'ok 4 # TODO') > xpass2.test
-echo 'Bail out!' > error.test
-(cat ok.test && echo 'Bail out!') > error2.test
+echo 'Bail out!' > bail.test
+
+(cat ok.test && echo 'Bail out!') > bail2.test
+
+cat > bail3.test <<'END'
+1..0 # SKIP
+Bail out!
+END
+
+# Too many tests.
+cat > error.test <<'END'
+1..2
+ok 1
+ok 2 # SKIP
+not ok 3
+not ok 4 # TODO
+END
+
+# Too few tests.
+cat > error2.test <<'END'
+1..4
+ok 1
+not ok 2 # TODO
+ok 3 # SKIP
+END
+
+# Repeated plan.
+cat > error3.test <<'END'
+1..2
+1..2
+ok 1
+ok 2
+END
+
+# Too many tests, after a "SKIP" plan.
+cat > error4.test <<'END'
+1..0 # SKIP
+ok 1
+ok 2
+END
+
+# Tests out of order.
+cat > error5.test <<'END'
+1..4
+not ok 1 # TODO
+ok 3
+ok 2
+ok 4
+END
+
+# Wrong test number.
+cat > error6.test <<'END'
+1..2
+ok 1 # SKIP
+ok 7
+END
+
+# No plan.
+cat > error7.test <<'END'
+ok 1 # SKIP
+ok 2 # TODO
+not ok 3 # TODO
+ok 4
+END
cat > hodgepodge.test <<'END'
1..2
@@ -76,17 +139,13 @@ ok 4 # TODO
Bail out!
END
-# TODO: add scripts with TAP errors (multiple plans, out-of-order
-# tests, etc).
-
tests=`echo *.test` # Also required later.
TESTS="$tests" $MAKE -e check >stdout && { cat stdout; Exit 1; }
cat stdout
-# Dirty tricks required here.
-for tst in ok skip skipall fail fail2 xpass xpass2 error error2 \
- hodgepodge hodgepodge-all; do
+# Dirty trick required here.
+for tst in `echo " $tests " | sed 's/.test / /'`; do
echo :copy-in-global-log: yes >> $tst.trs
done
@@ -112,8 +171,16 @@ have_rst_section 'FAIL: fail'
have_rst_section 'FAIL: fail2'
have_rst_section 'FAIL: xpass'
have_rst_section 'FAIL: xpass2'
+have_rst_section 'ERROR: bail'
+have_rst_section 'ERROR: bail2'
+have_rst_section 'ERROR: bail3'
have_rst_section 'ERROR: error'
have_rst_section 'ERROR: error2'
+have_rst_section 'ERROR: error3'
+have_rst_section 'ERROR: error4'
+have_rst_section 'ERROR: error5'
+have_rst_section 'ERROR: error6'
+have_rst_section 'ERROR: error7'
have_rst_section 'ERROR: hodgepodge'
have_rst_section 'ERROR: hodgepodge-all'
diff --git a/tests/tap-no-disable-hard-error.test
b/tests/tap-missing-plan-and-bad-exit.test
similarity index 65%
copy from tests/tap-no-disable-hard-error.test
copy to tests/tap-missing-plan-and-bad-exit.test
index 678e184..48c6fcc 100755
--- a/tests/tap-no-disable-hard-error.test
+++ b/tests/tap-missing-plan-and-bad-exit.test
@@ -15,36 +15,29 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
# TAP support:
-# - "Bail out!" magic and TAP parse errors are not disabled nor turned
-# into simple failures by the definition DISABLE_HARD_ERRORS.
+# - if a test script exits with non-zero status before printing the TAP
+# plan, then the driver reports both "missing plan" and "exited with
+# non-zero status" errors.
parallel_tests=yes
. ./defs || Exit 1
-cat > Makefile.am << 'END'
-DISABLE_HARD_ERRORS = yes
-TEST_LOG_COMPILER = cat
-TESTS = bail.test few.test noplan.test
+cat > foo.test <<END
+#!/bin/sh
+exit 123
END
-. "$testsrcdir"/tap-setup.sh || fatal_ "sourcing tap-setup.sh"
-
-cat > bail.test <<END
-1..1
-Bail out!
-END
+echo TESTS = foo.test > Makefile.am
-cat > few.test <<END
-1..1
-END
+chmod a+x foo.test
-cat > noplan.test <<END
-# nothing here
-END
+. "$testsrcdir"/tap-setup.sh || fatal_ "sourcing tap-setup.sh"
$MAKE check >stdout && { cat stdout; Exit 1; }
cat stdout
+count_test_results total=2 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=2
-count_test_results total=3 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=3
+grep '^ERROR: foo\.test - exited with status 123$' stdout
+grep '^ERROR: foo\.test - missing test plan$' stdout
:
diff --git a/tests/tap-plan-corner.test b/tests/tap-plan-corner.test
index eaebcf2..a818344 100755
--- a/tests/tap-plan-corner.test
+++ b/tests/tap-plan-corner.test
@@ -22,6 +22,8 @@ parallel_tests=yes
. "$testsrcdir"/tap-setup.sh || fatal_ "sourcing tap-setup.sh"
+# -------------------------------------------------------------------------
+
cat > leading-repeated.test <<END
1..1
1..1
@@ -42,6 +44,8 @@ for pos in leading trailing; do
grep "^ERROR: $pos-repeated\\.test - multiple test plans$" stdout
done
+# -------------------------------------------------------------------------
+
cat > leading-repeated.test <<END
1..2
ok 1
@@ -59,12 +63,44 @@ END
env TESTS="leading-repeated.test trailing-repeated.test" \
$MAKE -e check >stdout && { cat stdout; Exit 1; }
cat stdout
-count_test_results total=6 pass=2 fail=0 xpass=0 xfail=0 skip=0 error=4
-for pos in leading trailing; do
- grep "^ERROR: $pos-repeated\\.test - multiple test plans$" stdout
- grep "^ERROR: $pos-repeated\\.test 2 # AFTER LATE PLAN$" stdout
+count_test_results total=6 pass=3 fail=0 xpass=0 xfail=0 skip=0 error=3
+grep "^ERROR: leading-repeated\\.test - multiple test plans$" stdout
+grep "^ERROR: trailing-repeated\\.test - multiple test plans$" stdout
+grep "^ERROR: trailing-repeated\\.test 2 # AFTER LATE PLAN$" stdout
+grep "leading .*AFTER LATE PLAN" stdout && Exit 1
+
+# -------------------------------------------------------------------------
+
+cat > 1.test <<END
+1..0
+1..0
+END
+
+cat > 2.test <<END
+1..0 # SKIP
+1..0
+END
+
+cat > 3.test <<END
+1..0
+1..0 # SKIP
+END
+
+cat > 4.test <<END
+1..0 # SKIP
+1..0 # SKIP
+END
+
+env TESTS="1.test 2.test 3.test 4.test" \
+ $MAKE -e check >stdout && { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=8 pass=0 fail=0 xpass=0 xfail=0 skip=4 error=4
+for i in 1 2 3 4; do
+ grep "^ERROR: $i\\.test - multiple test plans$" stdout
done
+# -------------------------------------------------------------------------
+
cat > all.test <<END
1..5
ok 1
@@ -79,17 +115,17 @@ END
$MAKE -e check >stdout && { cat stdout; Exit 1; }
cat stdout
-count_test_results total=8 pass=2 fail=0 xpass=0 xfail=0 skip=0 error=6
+count_test_results total=8 pass=5 fail=0 xpass=0 xfail=0 skip=0 error=3
cat > exp <<'END'
PASS: all.test 1
PASS: all.test 2
ERROR: all.test - multiple test plans
-ERROR: all.test 3 # AFTER LATE PLAN
+PASS: all.test 3
ERROR: all.test - multiple test plans
-ERROR: all.test 4 # AFTER LATE PLAN
+PASS: all.test 4
ERROR: all.test - multiple test plans
-ERROR: all.test 5 # AFTER LATE PLAN
+PASS: all.test 5
END
$FGREP ': all.test' stdout > got
@@ -98,17 +134,55 @@ cat exp
cat got
diff exp got
-sed -e 1d all.test > t
-mv -f t all.test
+# -------------------------------------------------------------------------
+
+cat > all.test <<END
+1..2
+ok 1
+ok 2
+1..3
+ok 3
+END
+
+$MAKE -e check >stdout && { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=5 pass=2 fail=0 xpass=0 xfail=0 skip=0 error=3
+
+cat > exp <<'END'
+PASS: all.test 1
+PASS: all.test 2
+ERROR: all.test - multiple test plans
+ERROR: all.test 3 # UNPLANNED
+ERROR: all.test - too many tests run (expected 2, got 3)
+END
+
+$FGREP ': all.test' stdout > got
+
+cat exp
+cat got
+diff exp got
+
+# -------------------------------------------------------------------------
+
+cat > all.test <<END
+ok 1
+ok 2
+ok 3
+1..5
+1..5
+ok 4
+1..5
+ok 5
+END
$MAKE -e check >stdout && { cat stdout; Exit 1; }
cat stdout
-count_test_results total=7 pass=2 fail=0 xpass=0 xfail=0 skip=0 error=5
+count_test_results total=7 pass=3 fail=0 xpass=0 xfail=0 skip=0 error=4
cat > exp <<'END'
PASS: all.test 1
PASS: all.test 2
-ERROR: all.test 3 # AFTER LATE PLAN
+PASS: all.test 3
ERROR: all.test - multiple test plans
ERROR: all.test 4 # AFTER LATE PLAN
ERROR: all.test - multiple test plans
diff --git a/tests/tap-plan-errors.test b/tests/tap-plan-errors.test
index 95cc640..9d8abcd 100755
--- a/tests/tap-plan-errors.test
+++ b/tests/tap-plan-errors.test
@@ -19,9 +19,9 @@
# - multiple test plans
# - missing test plan
# - misplaced test plan (tests run after a late plan)
-# Checks about unplanned tests are performed in 'tap-unplanned.test'.
-# More checks about corner-cases in TAP plans are performed in
-# 'tap-plan-corner.test' and 'tap-plan-corner2.test'.
+# Checks about unplanned tests are performed in 'tap-unplanned.test'
+# and 'tap-skip-whole-badcount.test'. More checks about corner-cases
+# in TAP plans are performed in 'tap-plan-corner.test'.
parallel_tests=yes
. ./defs || Exit 1
@@ -76,6 +76,12 @@ ok 4
ok 5
END
+err='- missing test plan'
+my_check total=2 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+ok 1
+END
+
+
# The two test plans here are deliberately equal.
err='- multiple test plans'
my_check total=3 pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
@@ -85,9 +91,24 @@ ok 2
1..2
END
-err='- missing test plan'
-my_check total=2 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+# The second plan is diagnosed as extra, and only the first one is
+# relevant w.r.t. the number of the expected test.
+err='- multiple test plans'
+my_check total=4 pass=3 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+1..3
ok 1
+ok 2
+1..2
+ok 3
+END
+
+# As above, in a slightly different situation.
+err='- multiple test plans'
+my_check total=3 pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+1..2
+ok 1
+ok 2
+1..4
END
:
diff --git a/tests/tap-planskip-and-logging.test
b/tests/tap-planskip-and-logging.test
new file mode 100755
index 0000000..621608b
--- /dev/null
+++ b/tests/tap-planskip-and-logging.test
@@ -0,0 +1,74 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - interactions between "TAP plan with SKIP" and logging of earlier or
+# later TAP or non-TAP text
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+. "$testsrcdir"/tap-setup.sh || fatal_ "sourcing tap-setup.sh"
+
+echo TEST_LOG_DRIVER_FLAGS = --comments >> Makefile
+
+cat > foo.test <<END
+1..0
+a non-TAP line
+# a comment
+END
+
+cat > foo2.test <<END
+a non-TAP line 2
+# a comment 2
+1..0
+END
+
+cat > bar.test <<END
+# an early comment
+an early non-TAP line
+ $tab
+1..0 # SKIP
+# a later comment
+a later non-TAP line
+END
+
+TESTS='foo.test foo2.test bar.test' $MAKE -e check >stdout \
+ || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=3 pass=0 fail=0 xpass=0 xfail=0 skip=3 error=0
+
+grep '^# foo\.test: a comment$' stdout
+grep '^# foo2\.test: a comment 2$' stdout
+grep '^# bar\.test: an early comment$' stdout
+grep '^# bar\.test: a later comment$' stdout
+
+cat foo.log
+cat foo2.log
+cat bar.log
+
+grep '^a non-TAP line$' foo.log
+grep '^# a comment$' foo.log
+grep '^a non-TAP line 2$' foo2.log
+grep '^# a comment 2$' foo2.log
+grep '^# an early comment' bar.log
+grep '^an early non-TAP line$' bar.log
+grep '^# a later comment' bar.log
+grep '^a later non-TAP line$' bar.log
+grep "^ $tab$" bar.log
+
+:
diff --git a/tests/tap-skip-whole-badexit.test b/tests/tap-planskip-badexit.test
similarity index 100%
rename from tests/tap-skip-whole-badexit.test
rename to tests/tap-planskip-badexit.test
diff --git a/tests/tap-not-ok-skip.test b/tests/tap-planskip-bailout.test
similarity index 75%
copy from tests/tap-not-ok-skip.test
copy to tests/tap-planskip-bailout.test
index f154d23..0048e00 100755
--- a/tests/tap-not-ok-skip.test
+++ b/tests/tap-planskip-bailout.test
@@ -15,25 +15,22 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
# TAP support:
-# - a "not ok # SKIP" line should count as a failure, for consistency
-# with the prove(1) utility.
+# - a "Bail out!" directive causes an hard error, even if coming after
+# a "SKIP plan" (e.g., "1..0 # SKIP").
parallel_tests=yes
. ./defs || Exit 1
. "$testsrcdir"/tap-setup.sh || fatal_ "sourcing tap-setup.sh"
-cat > all.test <<'END'
-1..4
-not ok # SKIP
-not ok 2 # SKIP
-not ok - foo # SKIP
-not ok 4 - bar # SKIP
+cat > all.test <<END
+1..0 # SKIP
+Bail out!
END
$MAKE check >stdout && { cat stdout; Exit 1; }
cat stdout
-
-count_test_results total=4 pass=0 fail=4 skip=0 xpass=0 xfail=0 error=0
+count_test_results total=2 pass=0 fail=0 xpass=0 xfail=0 skip=1 error=1
+grep '^ERROR: all\.test - Bail out!' stdout
:
diff --git a/tests/tap-no-spurious-summary.test b/tests/tap-planskip-late.test
similarity index 59%
copy from tests/tap-no-spurious-summary.test
copy to tests/tap-planskip-late.test
index bc7fd51..c2cb0a8 100755
--- a/tests/tap-no-spurious-summary.test
+++ b/tests/tap-planskip-late.test
@@ -15,33 +15,30 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
# TAP support:
-# - `:test-results:' directives in test scripts' output doesn't
-# originate spurious results in the testsuite summary
+# - the special "plan with SKIP" can also be used "late" in the TAP
+# stream, i.e., preceded by non-TAP output or TAP diagnostic.
parallel_tests=yes
. ./defs || Exit 1
. "$testsrcdir"/tap-setup.sh || fatal_ "sourcing tap-setup.sh"
-cat > all.test <<'END'
-1..1
-:test-result: PASS
-:test-result: FAIL
-:test-result: XPASS
-:test-result: XFAIL
-:test-result: SKIP
-:test-result: ERROR
-:test-result: UNKNOWN
-ok 1
+cat > foo.test <<END
+some non-TAP text, will be copied in the global log
+1..0 # SKIP from the last line
END
-$MAKE check >stdout || { cat stdout; Exit 1; }
-cat stdout
+cat > bar.test <<END
+# some TAP diagnostic, will go to console
+1..0
+END
-count_test_results total=1 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=0
+TESTS='foo.test bar.test' $MAKE -e check >stdout || { cat stdout; Exit 1; }
+cat stdout
-for result in PASS FAIL XPASS XFAIL SKIP ERROR UNKNOWN; do
- grep "^ *:test-result: $result$" all.log
-done
+grep '^SKIP: foo\.test .* from the last line$' stdout
+grep '^SKIP: bar\.test$' stdout
+test `grep -c ': .*\.test' stdout` -eq 2
+count_test_results total=2 pass=0 fail=0 xpass=0 xfail=0 skip=2 error=0
:
diff --git a/tests/tap-not-ok-skip.test b/tests/tap-planskip-later-errors.test
similarity index 75%
copy from tests/tap-not-ok-skip.test
copy to tests/tap-planskip-later-errors.test
index f154d23..eb2ee26 100755
--- a/tests/tap-not-ok-skip.test
+++ b/tests/tap-planskip-later-errors.test
@@ -15,25 +15,26 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
# TAP support:
-# - a "not ok # SKIP" line should count as a failure, for consistency
-# with the prove(1) utility.
+# - TAP errors following a "TAP plan with SKIP" are still diagnosed.
parallel_tests=yes
. ./defs || Exit 1
. "$testsrcdir"/tap-setup.sh || fatal_ "sourcing tap-setup.sh"
-cat > all.test <<'END'
-1..4
-not ok # SKIP
-not ok 2 # SKIP
-not ok - foo # SKIP
-not ok 4 - bar # SKIP
+cat > all.test <<END
+1..0
+a non-TAP line
+1..3
+1..5
+# a comment
+1..1
END
$MAKE check >stdout && { cat stdout; Exit 1; }
cat stdout
-count_test_results total=4 pass=0 fail=4 skip=0 xpass=0 xfail=0 error=0
+count_test_results total=4 pass=0 fail=0 xpass=0 xfail=0 skip=1 error=3
+test `grep -c '^ERROR: all\.test - multiple test plans' stdout` -eq 3
:
diff --git a/tests/tap-no-disable-hard-error.test
b/tests/tap-planskip-unplanned-corner.test
similarity index 56%
copy from tests/tap-no-disable-hard-error.test
copy to tests/tap-planskip-unplanned-corner.test
index 678e184..cf9f49e 100755
--- a/tests/tap-no-disable-hard-error.test
+++ b/tests/tap-planskip-unplanned-corner.test
@@ -15,36 +15,33 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
# TAP support:
-# - "Bail out!" magic and TAP parse errors are not disabled nor turned
-# into simple failures by the definition DISABLE_HARD_ERRORS.
+# - test results seen in a TAP stream that has a "plan with SKIP" are
+# flagged as errors, even if all of them are "SKIP" results.
parallel_tests=yes
. ./defs || Exit 1
-cat > Makefile.am << 'END'
-DISABLE_HARD_ERRORS = yes
-TEST_LOG_COMPILER = cat
-TESTS = bail.test few.test noplan.test
-END
-
. "$testsrcdir"/tap-setup.sh || fatal_ "sourcing tap-setup.sh"
-cat > bail.test <<END
-1..1
-Bail out!
+cat > foo.test <<END
+1..0 # SKIP
+ok 1 # SKIP
END
-cat > few.test <<END
-1..1
+cat > bar.test <<END
+ok 1 # SKIP
+1..0 # SKIP
END
-cat > noplan.test <<END
-# nothing here
-END
-
-$MAKE check >stdout && { cat stdout; Exit 1; }
+TESTS='foo.test bar.test' $MAKE -e check >stdout && { cat stdout; Exit 1; }
cat stdout
-count_test_results total=3 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=3
+count_test_results total=5 pass=0 fail=0 xpass=0 xfail=0 skip=2 error=3
+
+grep '^SKIP: foo\.test$' stdout
+grep '^ERROR: foo\.test 1 # UNPLANNED$' stdout
+grep '^ERROR: foo\.test - too many tests run (expected 0, got 1)$' stdout
+grep '^SKIP: bar\.test 1 # SKIP$' stdout
+grep '^ERROR: bar\.test - too many tests run (expected 0, got 1)$' stdout
:
diff --git a/tests/tap-planskip-unplanned.test
b/tests/tap-planskip-unplanned.test
new file mode 100755
index 0000000..6286dac
--- /dev/null
+++ b/tests/tap-planskip-unplanned.test
@@ -0,0 +1,73 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - a "plan with skip" given after one or more test result have already
+# been seen is an error
+# - any test result following a "plan with skip" is an error.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+. "$testsrcdir"/tap-setup.sh || fatal_ "sourcing tap-setup.sh"
+
+cat > all.test <<END
+ok 1
+1..0 # SKIP too late
+END
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=2 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=1
+grep '^ERROR: all\.test - too many tests run (expected 0, got 1)$' stdout
+
+cat > all.test <<END
+ok 1
+ok 2 # SKIP
+not ok 3 # TODO
+1..0 # SKIP too late
+END
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=4 pass=1 fail=0 xpass=0 xfail=1 skip=1 error=1
+grep '^ERROR: all\.test - too many tests run (expected 0, got 3)$' stdout
+
+cat > all.test <<END
+1..0 # SKIP falsified later
+ok 1
+END
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=3 pass=0 fail=0 xpass=0 xfail=0 skip=1 error=2
+grep '^ERROR: all\.test 1 # UNPLANNED$' stdout
+grep '^ERROR: all\.test - too many tests run (expected 0, got 1)$' stdout
+
+cat > all.test <<END
+1..0 # SKIP falsified later
+ok 1
+ok 2 # SKIP
+not ok 3
+not ok 4 # TODO
+END
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=6 pass=0 fail=0 xpass=0 xfail=0 skip=1 error=5
+grep '^ERROR: all\.test 1 # UNPLANNED$' stdout
+grep '^ERROR: all\.test 2 # UNPLANNED$' stdout
+grep '^ERROR: all\.test 3 # UNPLANNED$' stdout
+grep '^ERROR: all\.test 4 # UNPLANNED$' stdout
+grep '^ERROR: all\.test - too many tests run (expected 0, got 4)$' stdout
+
+:
diff --git a/tests/tap-skip-whole-whitespace.test
b/tests/tap-planskip-whitespace.test
similarity index 100%
rename from tests/tap-skip-whole-whitespace.test
rename to tests/tap-planskip-whitespace.test
diff --git a/tests/tap-skip-whole.test b/tests/tap-planskip.test
similarity index 89%
rename from tests/tap-skip-whole.test
rename to tests/tap-planskip.test
index 6b14be4..2852305 100755
--- a/tests/tap-skip-whole.test
+++ b/tests/tap-planskip.test
@@ -26,44 +26,35 @@ weirdchars=\''"$!&()[]<>#;^?*/@%=,.:'
cat > foo.test <<END
1..0
-not ok 1
-not ok 2
END
cat > bar.test <<END
+blah
+# blah
1..0$tab $tab
-ok 1
-Bail out! some random failure
END
# It is undefined whether the comment after the plan below should
# count as an explanation; still, the test should be skipped.
cat > baz.test <<END
1..0 # WWW::Mechanize not installed
-ok 1
+other
+ junk
+ lines
END
cat > wget.test <<END
1..0 # SKIP wget(1) not installed
-not ok 1 # TODO
-ok 2
-ok 3 # SKIP
+# See also curl.test
END
cat > curl.test <<END
1..0 # skip: Can't connect to gnu.org!
-not ok 1
-ok 2 # TODO
+# See also wget.test
END
cat > mu.test <<END
1..0 # Skip $weirdchars
-# Various TAP errors in here shouldn't be seen, as the test is skipped.
-1..2
-ok 1
-ok 22
-not ok 333
-1..7
END
env TESTS='foo.test bar.test baz.test wget.test curl.test mu.test' \
diff --git a/tests/tap-skip-whole-lastline.test
b/tests/tap-skip-whole-lastline.test
deleted file mode 100755
index 25697e0..0000000
--- a/tests/tap-skip-whole-lastline.test
+++ /dev/null
@@ -1,59 +0,0 @@
-#! /bin/sh
-# Copyright (C) 2011 Free Software Foundation, Inc.
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; either version 2, or (at your option)
-# any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program. If not, see <http://www.gnu.org/licenses/>.
-
-# TAP support:
-# - the special plan format to skip all the tests in a script can also
-# be used on the last line, even if preceded by non-TAP output, TAP
-# diagnostic, and even non-failing TAP test results.
-
-parallel_tests=yes
-. ./defs || Exit 1
-
-. "$testsrcdir"/tap-setup.sh || fatal_ "sourcing tap-setup.sh"
-
-echo AM_TEST_LOG_DRIVER_FLAGS = --comments >> Makefile
-
-cat > foo.test <<END
-some non-TAP text, will be copied in the global log
-1..0 # SKIP from the last line
-END
-
-cat > bar.test <<END
-# some TAP diagnostic, will go to console
-1..0
-END
-
-cat > baz.test <<END
-ok 1 - seen also if the whole test is skipped!
-1..0 # SKIP
-END
-
-st=0
-TESTS='foo.test bar.test baz.test' $MAKE -e check >stdout || st=$?
-cat stdout
-cat test-suite.log
-test $st -eq 0
-
-grep '^SKIP: foo\.test .* from the last line$' stdout
-grep '^SKIP: bar\.test$' stdout
-grep '^SKIP: baz\.test$' stdout
-grep '^# bar\.test: some TAP diagnostic, will go to console$' stdout
-grep '^some non-TAP text, will be copied in the global log$' test-suite.log
-grep '^PASS: baz\.test 1 - seen also if the whole test is skipped!$' stdout
-test `grep -c ': .*\.test' stdout` -eq 4
-count_test_results total=4 pass=1 fail=0 xpass=0 xfail=0 skip=3 error=0
-
-:
diff --git a/tests/tap-todo-skip-whitespace.test
b/tests/tap-todo-skip-whitespace.test
index 695e98c..fe32c97 100755
--- a/tests/tap-todo-skip-whitespace.test
+++ b/tests/tap-todo-skip-whitespace.test
@@ -76,7 +76,7 @@ my_make_check ()
# Don't be too strict w.r.t. possible normalization of "TODO: foo" into
# "TODO : foo" (as is done by, e.g., the 'TAP::Parser' perl module).
LC_ALL=C grep '^[A-Z][A-Z]*:' stdout \
- | sed -e 's/# \(TODO\|SKIP\) *:/# \1:/' > got
+ | sed -e 's/# TODO *:/# TODO:/' -e 's/# SKIP *:/# SKIP:/' > got
cat exp
cat got
diff exp got
diff --git a/tests/test-driver-custom-no-html.test
b/tests/test-driver-custom-no-html.test
index 2c86026..37b93bc 100755
--- a/tests/test-driver-custom-no-html.test
+++ b/tests/test-driver-custom-no-html.test
@@ -62,6 +62,8 @@ $FGREP 'dummy title' test-suite.log
# Sanity check: trying to produce HTML output should fail.
$MAKE check-html >output 2>&1 && { cat output; Exit 1; }
cat output
-$FGREP SEVERE output
+# The second grep is to avoid spurious failures on systems without
+# a working `rst2html' program.
+$FGREP SEVERE output || $FGREP 'cannot find rst2html' output
:
hooks/post-receive
--
GNU Automake
| [Prev in Thread] |
Current Thread |
[Next in Thread] |
- [Automake-commit] [SCM] GNU Automake branch, testsuite-work, updated. v1.11-1199-g66dc614,
Stefano Lattarini <=