[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Automake-commit] [SCM] GNU Automake branch, test-protocols, updated. v1
|
From: |
Stefano Lattarini |
|
Subject: |
[Automake-commit] [SCM] GNU Automake branch, test-protocols, updated. v1.11-1017-g9e59677 |
|
Date: |
Fri, 12 Aug 2011 16:29:02 +0000 |
This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "GNU Automake".
http://git.sv.gnu.org/gitweb/?p=automake.git;a=commitdiff;h=9e5967708814027981d910e6fed4f7789eba23f8
The branch, test-protocols has been updated
via 9e5967708814027981d910e6fed4f7789eba23f8 (commit)
via c1d0de60cd864a0bf27df370a9d01640944ed8e2 (commit)
via e7ca7771e57e0d02c9dedf0e3bfc5128b397d7f2 (commit)
via 52a130494d747f224f4d6cc7a487942179857135 (commit)
via bf69289c8b6f060de7946fe455c12c2fbdddd053 (commit)
via 320685333ede68b51e2c720941b4f90010268760 (commit)
via d4a752a353c3fd15dbe7cb5d2b5e01996546d75c (commit)
via cc485aba6babbe23c0b67eb28eb6181f18fedb98 (commit)
via d35cee2a6c394e13b81ebef29fe53eea4e2b533d (commit)
via b2d6b49c50859a4b0773f6e20756e2ece20f092c (commit)
from 7fff2880a5e284f17cd6f0d869b0addeef7e1f55 (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
- Log -----------------------------------------------------------------
commit 9e5967708814027981d910e6fed4f7789eba23f8
Author: Stefano Lattarini <address@hidden>
Date: Fri Aug 12 17:17:48 2011 +0200
fix: two recent ChangeLog entries
commit c1d0de60cd864a0bf27df370a9d01640944ed8e2
Author: Stefano Lattarini <address@hidden>
Date: Fri Aug 12 17:04:33 2011 +0200
tap: improve diagnosing and reporting of plan mismatches
Problem exposed by a failure in the test 'tap-plan-errors.test'.
* lib/tap-driver (NO_PLAN, EARLY_PLAN, LATE_PLAN): New constants.
(Throughout the file): Use more complex semantics for `$plan_seen',
which now also remember whether the plan that has been seen is an
"early" or "late" plan; in turn, this makes ...
($tap_stopped): ... this variable redundant; remove it.
(handle_tap_plan): Adjust to avoid producing spurious or confusing
error messages about extra or mismatched "late" TAP plans.
* tests/tap-plan-corned.test: Adjust and extend.
* tests/tap-color.test: Likewise.
commit e7ca7771e57e0d02c9dedf0e3bfc5128b397d7f2
Author: Stefano Lattarini <address@hidden>
Date: Fri Aug 12 15:51:12 2011 +0200
tap: log all TAP stream, even after a "Bail out!"
* lib/tap-driver ($bailed_out): New global boolean variable,
telling whether a "Bail out!" directive has been seen or not.
(handle_tap_bailout): This function does not anymore stop the
reading from TAP stream; instead, it sets `$bailed_out' to a
true value, so that only the subsequent parsing of the input
TAP stream is stopped.
(finish): Remove, no more needed, its contents inlined into ...
(main): ... this function, with related adjustments in the code
flow.
(get_test_exit_message): Do not "flush" the input TAP stream
to fetch the exit status of test script, it is not anymore
required. Add a sanity check.
* tests/tap-bailout-and-logging.test: New test.
* tests/Makefile.am (tap_with_common_setup_tests): Update.
commit 52a130494d747f224f4d6cc7a487942179857135
Author: Stefano Lattarini <address@hidden>
Date: Fri Aug 12 14:42:21 2011 +0200
coverage: TAP diagnostics after "Bail out!" aren't reported
This is compatible with the behaviour of the `prove' utility.
* tests/tap-bailout-suppress-later-diagnostic.test: New test.
* tests/Makefile.am (tap_with_common_setup_tests): Update.
commit bf69289c8b6f060de7946fe455c12c2fbdddd053
Author: Stefano Lattarini <address@hidden>
Date: Fri Aug 12 13:38:34 2011 +0200
coverage: TAP errors after a "Bail out!" directive aren't reported
This is compatible with the behaviour of the `prove' utility.
* tests/tap-bailout-suppress-later-errors.test: New test.
* tests/tap-bailout-badexit.test: Renamed ...
* tests/tap-bailout-suppress-badexit.test: ... to this, for
consistency.
* tests/Makefile.am (tap_with_common_setup_tests): Update.
commit 320685333ede68b51e2c720941b4f90010268760
Author: Stefano Lattarini <address@hidden>
Date: Fri Aug 12 13:12:54 2011 +0200
testsuite: reorganize tests on TAP plans
* tests/tap-skip-whole.test: Rename ...
* tests/tap-planskip.test: ... to this
* tests/tap-skip-whole-badexit.test: Rename ...
* tests/tap-planskip-badexit.test: ... to this.
* tests/tap-skip-whole-bailout.test: Rename ...
* tests/tap-planskip-bailout.test: ... to this.
* tests/tap-skip-whole-bailout.test: Rename ...
* tests/tap-planskip-later-errors.test: ... to this.
* tests/tap-skip-whole-unplanned.test: Rename ...
* tests/tap-planskip-unplanned.test: ... to this.
* tests/tap-skip-whole-whitespace.test: Rename ...
* tests/tap-planskip-whitespace.test: ... to this.
* tests/tap-skip-whole-continue.test: Rename ...
* tests/tap-planskip-and-logging.test: ... to this, extend to try
more cases, and fix heading comments.
* tests/tap-skip-whole-lastline.test: Rename ...
* tests/tap-planskip-late.test: ... to this, fix heading comments,
and remove some checks that are just a duplication of those already
performed in the previous script.
* tests/tap-results-with-planskip.test: Rename to ...
* tests/tap-planskip-unplanned-corner.test: ... this, and improve
hading comments.
* tests/tap-planskip-later-errors.test: New test.
* tests/Makefile.am (tap_with_common_setup_tests): Update.
commit d4a752a353c3fd15dbe7cb5d2b5e01996546d75c
Author: Stefano Lattarini <address@hidden>
Date: Fri Aug 12 11:38:55 2011 +0200
coverage: more TAP driver consistency w.r.t. 'prove'
* tests/skip-results-with-planskip.test: New test, verifies that
test results seen in a TAP stream that has a "plan with SKIP" are
flagged as errors.
* tests/Makefile.am (tap_with_common_setup_tests): Update.
commit cc485aba6babbe23c0b67eb28eb6181f18fedb98
Author: Stefano Lattarini <address@hidden>
Date: Fri Aug 12 11:03:46 2011 +0200
tap: improve some comments in the TAP driver
* lib/tap-driver (handle_tap_plan): Improve comments describing
possible errors in a "plan with SKIP" directive.
commit d35cee2a6c394e13b81ebef29fe53eea4e2b533d
Author: Stefano Lattarini <address@hidden>
Date: Fri Aug 12 10:49:05 2011 +0200
tap: non-zero exit status after "Bail out!" should not be reported
This new behaviour is more consistent with that of the `prove'
utility. Also, experience with the use of TAP in the Automake's
own test suite has shown that reporting non-zero exit status from
a script after it has issued a "Bail out!" directive is mostly
redundant, more introducing noise rather than conveying useful
information.
* lib/tap-driver (finish): Inline the part of it that was getting
the script exit status ...
(main): ... in here.
* tests/tap-bailout-badexit.test: New test.
* tests/Makefile.am (tap_with_common_setup_tests): Update.
commit b2d6b49c50859a4b0773f6e20756e2ece20f092c
Author: Stefano Lattarini <address@hidden>
Date: Wed Aug 10 15:36:58 2011 +0200
tap: a "plan with SKIP" after test results is an error
This new behaviour is both more natural and more consistent with
that of the `prove' utility.
* lib/tap-driver (handle_tap_plan): Do not stop TAP parsing when
a "plan with SKIP" line (e.g., "1..0 # SKIP") is encountered, and
do not print a "SKIP" line if some TAP result has already been
seen.
* tests/tap-skip-whole.test: Adapt.
* tests/tap-skip-whole-lastline.test: Likewise.
* tests/tap-global-result.test: Adapt and extend.
* tests/tap-skip-plan-errors.test: Likewise, and fix an obsolete
small part of a comment.
* tests/tap-skip-whole-bailout.test: New test.
* tests/tap-skip-whole-unplanned.test: Likewise.
* tests/tap-skip-whole-continue.test: Likewise.
* tests/Makefile.am (tap_with_common_setup_tests): Update.
-----------------------------------------------------------------------
Summary of changes:
ChangeLog | 137 ++++++++++++++++++-
lib/tap-driver | 112 ++++++++++-------
tests/Makefile.am | 17 ++-
tests/Makefile.in | 17 ++-
...-together.test => tap-bailout-and-logging.test} | 32 +++--
...exit.test => tap-bailout-suppress-badexit.test} | 45 ++++---
... => tap-bailout-suppress-later-diagnostic.test} | 29 ++---
tests/tap-bailout-suppress-later-errors.test | 74 +++++++++++
tests/tap-color.test | 10 +-
tests/tap-global-result.test | 85 +++++++++++--
tests/tap-plan-corner.test | 98 ++++++++++++--
tests/tap-plan-errors.test | 31 ++++-
tests/tap-planskip-and-logging.test | 74 +++++++++++
...hole-badexit.test => tap-planskip-badexit.test} | 0
...-not-ok-skip.test => tap-planskip-bailout.test} | 17 +--
...purious-summary.test => tap-planskip-late.test} | 33 ++---
...ok-skip.test => tap-planskip-later-errors.test} | 19 ++--
...ror.test => tap-planskip-unplanned-corner.test} | 35 +++---
tests/tap-planskip-unplanned.test | 73 +++++++++++
...hitespace.test => tap-planskip-whitespace.test} | 0
tests/{tap-skip-whole.test => tap-planskip.test} | 23 +---
tests/tap-skip-whole-lastline.test | 59 ---------
22 files changed, 755 insertions(+), 265 deletions(-)
copy tests/{tap-todo-skip-together.test => tap-bailout-and-logging.test} (61%)
copy tests/{tap-skip-whole-badexit.test => tap-bailout-suppress-badexit.test}
(56%)
copy tests/{tap-no-disable-hard-error.test =>
tap-bailout-suppress-later-diagnostic.test} (67%)
create mode 100755 tests/tap-bailout-suppress-later-errors.test
create mode 100755 tests/tap-planskip-and-logging.test
rename tests/{tap-skip-whole-badexit.test => tap-planskip-badexit.test} (100%)
copy tests/{tap-not-ok-skip.test => tap-planskip-bailout.test} (75%)
copy tests/{tap-no-spurious-summary.test => tap-planskip-late.test} (59%)
copy tests/{tap-not-ok-skip.test => tap-planskip-later-errors.test} (75%)
copy tests/{tap-no-disable-hard-error.test =>
tap-planskip-unplanned-corner.test} (56%)
create mode 100755 tests/tap-planskip-unplanned.test
rename tests/{tap-skip-whole-whitespace.test => tap-planskip-whitespace.test}
(100%)
rename tests/{tap-skip-whole.test => tap-planskip.test} (89%)
delete mode 100755 tests/tap-skip-whole-lastline.test
diff --git a/ChangeLog b/ChangeLog
index 4e50d37..8b4ab34 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,128 @@
+2011-08-12 Stefano Lattarini <address@hidden>
+
+ tap: improve diagnosing and reporting of plan mismatches
+ Problem exposed by a failure in the test 'tap-plan-errors.test'.
+ * lib/tap-driver (NO_PLAN, EARLY_PLAN, LATE_PLAN): New constants.
+ (Throughout the file): Use more complex semantics for `$plan_seen',
+ which now also remember whether the plan that has been seen is an
+ "early" or "late" plan; in turn, this makes ...
+ ($tap_stopped): ... this variable redundant; remove it.
+ (handle_tap_plan): Adjust to avoid producing spurious or confusing
+ error messages about extra or mismatched "late" TAP plans.
+ * tests/tap-plan-corned.test: Adjust and extend.
+ * tests/tap-color.test: Likewise.
+
+2011-08-12 Stefano Lattarini <address@hidden>
+
+ tap: log all TAP stream, even after a "Bail out!"
+ * lib/tap-driver ($bailed_out): New global boolean variable,
+ telling whether a "Bail out!" directive has been seen or not.
+ (handle_tap_bailout): This function does not anymore stop the
+ reading from TAP stream; instead, it sets `$bailed_out' to a
+ true value, so that only the subsequent parsing of the input
+ TAP stream is stopped.
+ (finish): Remove, no more needed, its contents inlined into ...
+ (main): ... this function, with related adjustments in the code
+ flow.
+ (get_test_exit_message): Do not "flush" the input TAP stream
+ to fetch the exit status of test script, it is not anymore
+ required. Add a sanity check.
+ * tests/tap-bailout-and-logging.test: New test.
+ * tests/Makefile.am (tap_with_common_setup_tests): Update.
+
+2011-08-12 Stefano Lattarini <address@hidden>
+
+ coverage: TAP diagnostics after "Bail out!" aren't reported
+ This is compatible with the behaviour of the `prove' utility.
+ * tests/tap-bailout-suppress-later-diagnostic.test: New test.
+ * tests/Makefile.am (tap_with_common_setup_tests): Update.
+
+2011-08-12 Stefano Lattarini <address@hidden>
+
+ coverage: TAP errors after a "Bail out!" directive aren't reported
+ This is compatible with the behaviour of the `prove' utility.
+ * tests/tap-bailout-suppress-later-errors.test: New test.
+ * tests/tap-bailout-badexit.test: Renamed ...
+ * tests/tap-bailout-suppress-badexit.test: ... to this, for
+ consistency.
+ * tests/Makefile.am (tap_with_common_setup_tests): Update.
+
+2011-08-12 Stefano Lattarini <address@hidden>
+
+ testsuite: reorganize tests on TAP plans
+ * tests/tap-skip-whole.test: Rename ...
+ * tests/tap-planskip.test: ... to this
+ * tests/tap-skip-whole-badexit.test: Rename ...
+ * tests/tap-planskip-badexit.test: ... to this.
+ * tests/tap-skip-whole-bailout.test: Rename ...
+ * tests/tap-planskip-bailout.test: ... to this.
+ * tests/tap-skip-whole-bailout.test: Rename ...
+ * tests/tap-planskip-later-errors.test: ... to this.
+ * tests/tap-skip-whole-unplanned.test: Rename ...
+ * tests/tap-planskip-unplanned.test: ... to this.
+ * tests/tap-skip-whole-whitespace.test: Rename ...
+ * tests/tap-planskip-whitespace.test: ... to this.
+ * tests/tap-skip-whole-continue.test: Rename ...
+ * tests/tap-planskip-and-logging.test: ... to this, extend to try
+ more cases, and fix heading comments.
+ * tests/tap-skip-whole-lastline.test: Rename ...
+ * tests/tap-planskip-late.test: ... to this, fix heading comments,
+ and remove some checks that are just a duplication of those already
+ performed in the previous script.
+ * tests/tap-results-with-planskip.test: Rename to ...
+ * tests/tap-planskip-unplanned-corner.test: ... this, and improve
+ hading comments.
+ * tests/tap-planskip-later-errors.test: New test.
+ * tests/Makefile.am (tap_with_common_setup_tests): Update.
+
+2011-08-12 Stefano Lattarini <address@hidden>
+
+ coverage: more TAP driver consistency w.r.t. 'prove'
+ * tests/skip-results-with-planskip.test: New test, verifies that
+ test results seen in a TAP stream that has a "plan with SKIP" are
+ flagged as errors.
+ * tests/Makefile.am (tap_with_common_setup_tests): Update.
+
+2011-08-12 Stefano Lattarini <address@hidden>
+
+ tap: improve some comments in the TAP driver
+ * lib/tap-driver (handle_tap_plan): Improve comments describing
+ possible errors in a "plan with SKIP" directive.
+
+2011-08-12 Stefano Lattarini <address@hidden>
+
+ tap: non-zero exit status after "Bail out!" should not be reported
+ This new behaviour is more consistent with that of the `prove'
+ utility. Also, experience with the use of TAP in the Automake's
+ own test suite has shown that reporting non-zero exit status from
+ a script after it has issued a "Bail out!" directive is mostly
+ redundant, more introducing noise rather than conveying useful
+ information.
+ * lib/tap-driver (finish): Inline the part of it that was getting
+ the script exit status ...
+ (main): ... in here.
+ * tests/tap-bailout-badexit.test: New test.
+ * tests/Makefile.am (tap_with_common_setup_tests): Update.
+
+2011-08-11 Stefano Lattarini <address@hidden>
+
+ tap: a "plan with SKIP" after test results is an error
+ This new behaviour is both more natural and more consistent with
+ that of the `prove' utility.
+ * lib/tap-driver (handle_tap_plan): Do not stop TAP parsing when
+ a "plan with SKIP" line (e.g., "1..0 # SKIP") is encountered, and
+ do not print a "SKIP" line if some TAP result has already been
+ seen.
+ * tests/tap-skip-whole.test: Adapt.
+ * tests/tap-skip-whole-lastline.test: Likewise.
+ * tests/tap-global-result.test: Adapt and extend.
+ * tests/tap-skip-plan-errors.test: Likewise, and fix an obsolete
+ small part of a comment.
+ * tests/tap-skip-whole-bailout.test: New test.
+ * tests/tap-skip-whole-unplanned.test: Likewise.
+ * tests/tap-skip-whole-continue.test: Likewise.
+ * tests/Makefile.am (tap_with_common_setup_tests): Update.
+
2011-08-09 Stefano Lattarini <address@hidden>
test harness: avoid possible fork bomb
@@ -6,17 +131,13 @@
* lib/am/check.am (am--redo-logs): Detect possible infinite
recursion due to a test log in $(TEST_LOGS) being the same
as $(TEST_SUITE_LOG).
- * tests/parallel-tests-fork-bomb.test: Enhance and extend a
- little now that we have more explicit semantics.
2011-08-09 Stefano Lattarini <address@hidden>
- coverage: possible infinite recursion in the test harness
- Motivated by a regression in the 'test-protocols' branch.
- * tests/parallel-tests-fork-bomb.test: New test, checking that
- if $(TEST_SUITE_LOG) is in $(TEST_LOGS), we obtain a diagnosed
- error rather than a make hang or a fork bomb.
- * tests/Makefile.am (TESTS): Update.
+ coverage: enhance test against infinite recursion in test harness
+ * tests/parallel-tests-fork-bomb.test: Enhance and extend a
+ little now, for soon-to-be-implemented more explicit and safer
+ semantics.
2011-08-09 Stefano Lattarini <address@hidden>
diff --git a/lib/tap-driver b/lib/tap-driver
index 2393346..95fb78c 100755
--- a/lib/tap-driver
+++ b/lib/tap-driver
@@ -39,17 +39,22 @@ my %COLOR = (
std => "\e[m",
);
+# It's important that NO_PLAN evaluates "false" as a boolean.
+use constant NO_PLAN => 0;
+use constant EARLY_PLAN => 1;
+use constant LATE_PLAN => 2;
+
# ------------------- #
# Global variables. #
# ------------------- #
my $testno = 0; # Number of test results seen so far.
-my $plan_seen = 0; # Whether the TAP plan has been seen or not.
+my $bailed_out = 0; # Whether a "Bail out!" directive has been seen.
my $parser; # TAP parser object (will be initialized later).
-# When true, it means that the rest of the input stream cannot
-# contain any further TAP results.
-my $tap_stopped = 0;
+# Whether the TAP plan has been seen or not, and if yes, which kind
+# it is ("early" is seen before any test result, "late" otherwise).
+my $plan_seen = NO_PLAN;
# ----------------- #
# Option parsing. #
@@ -96,7 +101,6 @@ sub colored ($$);
sub copy_in_global_log ();
sub decorate_result ($);
sub extract_tap_comment ($);
-sub finish ();
sub get_global_test_result ();
sub get_test_exit_message ();
sub get_test_results ();
@@ -210,10 +214,10 @@ sub start (@)
sub get_test_exit_message ()
{
- # Flush all the remaining TAP stream, so that we can obtain the
- # exit status of the TAP producer.
- do {} while defined $parser->next;
my $wstatus = $parser->wait;
+ # Watch out for possible internal errors.
+ die "couldn't get the exit ststus of the TAP producer"
+ unless defined $wstatus;
# Return an undefined value if the producer exited with success.
return unless $wstatus;
# Otherwise, determine whether it exited with error or was terminated
@@ -233,23 +237,14 @@ sub get_test_exit_message ()
}
}
-sub finish ()
-{
- if (!$cfg{"ignore-exit"} and my $msg = get_test_exit_message)
- {
- testsuite_error $msg;
- }
- write_test_results;
- close LOG or die "closing $log_file: $!\n";
- exit 0;
-}
-
sub stringify_test_result ($)
{
my $result = shift;
my $PASS = $cfg{"expect-failure"} ? "XPASS": "PASS";
my $FAIL = $cfg{"expect-failure"} ? "XFAIL": "FAIL";
- if ($result->is_unplanned || $result->number != $testno || $tap_stopped)
+ if ($result->is_unplanned
+ || $result->number != $testno
+ || $plan_seen == LATE_PLAN)
{
return "ERROR";
}
@@ -340,7 +335,7 @@ sub handle_tap_test ($)
$string .= " $description";
}
- if ($tap_stopped)
+ if ($plan_seen == LATE_PLAN)
{
$string .= " # AFTER LATE PLAN";
}
@@ -367,28 +362,38 @@ sub handle_tap_test ($)
sub handle_tap_plan ($)
{
my $plan = shift;
- # Only one plan per stream is acceptable.
- testsuite_error "multiple test plans" if $plan_seen;
- $plan_seen = 1;
- # TAP plan must come either before or after *all* the TAP results.
- # So, if we find it after having already seen at least one TAP result,
- # set a flag signaling that no more TAP results are acceptable.
- $tap_stopped = 1 if $testno >= 1;
- # Nothing more to do, unless the plan contains a SKIP directive.
- return
- if not defined $plan->directive && length ($plan->directive) > 0;
- my $explanation = $plan->explanation ?
- "- " . $plan->explanation : undef;
- report "SKIP", $explanation;
- finish;
+ if ($plan_seen)
+ {
+ # Error, only one plan per stream is acceptable.
+ testsuite_error "multiple test plans";
+ return;
+ }
+ # The TAP plan can come before or after *all* the TAP results; we speak
+ # respectively of an "early" or a "late" plan. If we see the plan line
+ # after at least one TAP result has been seen, assume we have a late
+ # plan; in this case, any further test result seen after the plan will
+ # be flagged as an error.
+ $plan_seen = ($testno >= 1 ? LATE_PLAN : EARLY_PLAN);
+ # If $testno > 0, we have an error ("too many tests run") that will be
+ # automatically dealt with later, so don't worry about it here. If
+ # $plan_seen is true, we have an error due to a repeated plan, and that
+ # has already been dealt with above. Otherwise, we have a valid "plan
+ # with SKIP" specification, and should report it as a particular kind
+ # of SKIP result.
+ if ($plan->directive && $testno == 0)
+ {
+ my $explanation = $plan->explanation ?
+ "- " . $plan->explanation : undef;
+ report "SKIP", $explanation;
+ }
}
sub handle_tap_bailout ($)
{
my ($bailout, $msg) = ($_[0], "Bail out!");
+ $bailed_out = 1;
$msg .= " " . $bailout->explanation if $bailout->explanation;
testsuite_error $msg;
- finish;
}
sub extract_tap_comment ($)
@@ -410,6 +415,9 @@ sub main (@)
{
# Verbatim copy any input line into the log file.
print $cur->raw . "\n";
+ # Parsing of TAP input should stop after a "Bail out!" directive.
+ next if $bailed_out;
+
if ($cur->is_plan)
{
handle_tap_plan ($cur);
@@ -428,18 +436,30 @@ sub main (@)
report "#", "$comment" if length $comment;
}
}
- if (!$plan_seen)
- {
- testsuite_error "missing test plan";
- }
- elsif ($parser->tests_planned != $parser->tests_run)
+ # A "Bail out!" directive should cause us to ignore any following TAP
+ # error, as well as a non-zero exit status from the TAP producer.
+ if (!$bailed_out)
{
- my ($planned, $run) = ($parser->tests_planned, $parser->tests_run);
- my $bad_amount = $run > $planned ? "many" : "few";
- testsuite_error (sprintf "too %s tests run (expected %d, got %d)",
- $bad_amount, $planned, $run);
+ if (!$plan_seen)
+ {
+ testsuite_error "missing test plan";
+ }
+ elsif ($parser->tests_planned != $parser->tests_run)
+ {
+ my ($planned, $run) = ($parser->tests_planned, $parser->tests_run);
+ my $bad_amount = $run > $planned ? "many" : "few";
+ testsuite_error (sprintf "too %s tests run (expected %d, got %d)",
+ $bad_amount, $planned, $run);
+ }
}
- finish;
+ if (!$cfg{"ignore-exit"} && !$bailed_out)
+ {
+ my $msg = get_test_exit_message ();
+ testsuite_error $msg if $msg;
+ }
+ write_test_results;
+ close LOG or die "closing $log_file: $!\n";
+ exit 0;
}
# ----------- #
diff --git a/tests/Makefile.am b/tests/Makefile.am
index bac1d32..7af1eb6 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -1144,6 +1144,10 @@ testsuite-summary-count-many.log:
extract-testsuite-summary
tap_with_common_setup_tests = \
tap-autonumber.test \
tap-bailout.test \
+tap-bailout-and-logging.test \
+tap-bailout-suppress-badexit.test \
+tap-bailout-suppress-later-diagnostic.test \
+tap-bailout-suppress-later-errors.test \
tap-color.test \
tap-deps.test \
tap-diagnostic.test \
@@ -1173,12 +1177,17 @@ tap-plan.test \
tap-plan-corner.test \
tap-plan-errors.test \
tap-plan-middle.test \
+tap-planskip.test \
+tap-planskip-late.test \
+tap-planskip-and-logging.test \
+tap-planskip-unplanned.test \
+tap-planskip-unplanned-corner.test \
+tap-planskip-whitespace.test \
+tap-planskip-badexit.test \
+tap-planskip-bailout.test \
+tap-planskip-later-errors.test \
tap-realtime.test \
tap-recheck-logs.test \
-tap-skip-whole-whitespace.test \
-tap-skip-whole.test \
-tap-skip-whole-lastline.test \
-tap-skip-whole-badexit.test \
tap-todo-skip-together.test \
tap-todo-skip-whitespace.test \
tap-todo-skip.test \
diff --git a/tests/Makefile.in b/tests/Makefile.in
index 9e03806..e294596 100644
--- a/tests/Makefile.in
+++ b/tests/Makefile.in
@@ -1384,6 +1384,10 @@ $(parallel_tests)
tap_with_common_setup_tests = \
tap-autonumber.test \
tap-bailout.test \
+tap-bailout-and-logging.test \
+tap-bailout-suppress-badexit.test \
+tap-bailout-suppress-later-diagnostic.test \
+tap-bailout-suppress-later-errors.test \
tap-color.test \
tap-deps.test \
tap-diagnostic.test \
@@ -1413,12 +1417,17 @@ tap-plan.test \
tap-plan-corner.test \
tap-plan-errors.test \
tap-plan-middle.test \
+tap-planskip.test \
+tap-planskip-late.test \
+tap-planskip-and-logging.test \
+tap-planskip-unplanned.test \
+tap-planskip-unplanned-corner.test \
+tap-planskip-whitespace.test \
+tap-planskip-badexit.test \
+tap-planskip-bailout.test \
+tap-planskip-later-errors.test \
tap-realtime.test \
tap-recheck-logs.test \
-tap-skip-whole-whitespace.test \
-tap-skip-whole.test \
-tap-skip-whole-lastline.test \
-tap-skip-whole-badexit.test \
tap-todo-skip-together.test \
tap-todo-skip-whitespace.test \
tap-todo-skip.test \
diff --git a/tests/tap-todo-skip-together.test
b/tests/tap-bailout-and-logging.test
similarity index 61%
copy from tests/tap-todo-skip-together.test
copy to tests/tap-bailout-and-logging.test
index 14b8ed5..9854ea8 100755
--- a/tests/tap-todo-skip-together.test
+++ b/tests/tap-bailout-and-logging.test
@@ -15,7 +15,8 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
# TAP support:
-# - TODO and SKIP directives on the same line: the first one wins
+# - even after a "Bail out!" directive, all input is still copied in
+# the log file
parallel_tests=yes
. ./defs || Exit 1
@@ -23,17 +24,26 @@ parallel_tests=yes
. "$testsrcdir"/tap-setup.sh || fatal_ "sourcing tap-setup.sh"
cat > all.test <<END
-1..2
-ok 1 # SKIP TODO
-not ok 2 # TODO SKIP
+First line
+Bail out!
+non-TAP line after bailout
+# TAP diagnostic after bailout
+1..0 # SKIP TAP plan after bailout
+ok 1 - TAP result after bailout
END
-$MAKE check >stdout || { cat stdout; Exit 1; }
-cat stdout
-
-count_test_results total=2 pass=0 fail=0 xpass=0 xfail=1 skip=1 error=0
-
-grep '^SKIP: all\.test 1 ' stdout
-grep '^XFAIL: all\.test 2 ' stdout
+$MAKE check && { cat all.log; Exit 1; }
+cat all.log
+
+for rx in \
+ 'First line' \
+ 'Bail out!' \
+ 'non-TAP line after bailout' \
+ '# TAP diagnostic after bailout' \
+ '1\.\.0 # SKIP TAP plan after bailout' \
+ 'ok 1 - TAP result after bailout' \
+; do
+ grep "^$rx$" all.log
+done
:
diff --git a/tests/tap-skip-whole-badexit.test
b/tests/tap-bailout-suppress-badexit.test
similarity index 56%
copy from tests/tap-skip-whole-badexit.test
copy to tests/tap-bailout-suppress-badexit.test
index 76ba692..059394c 100755
--- a/tests/tap-skip-whole-badexit.test
+++ b/tests/tap-bailout-suppress-badexit.test
@@ -14,39 +14,52 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
-# TAP support:
-# - an exit status != 0 of a test script causes an hard error, even if
-# the last line of output is a "SKIP plan" (e.g., "1..0 # SKIP").
+# Basic TAP test protocol support:
+# - A "Bail out!" directive causes the driver to ignore the exit
+# status of the test script.
parallel_tests=yes
. ./defs || Exit 1
-echo TESTS = one.test two.test > Makefile.am
+tests='exit.test exit127.test sighup.test sigterm.test'
+
+cat > Makefile.am <<END
+TESTS = $tests
+END
. "$testsrcdir"/tap-setup.sh || fatal_ "sourcing tap-setup.sh"
-cat > one.test <<'END'
+cat > exit.test << 'END'
#!/bin/sh
-echo '1..0 # SKIP'
+echo 'Bail out!'
exit 1
END
-cat > two.test <<'END'
+cat > exit127.test << 'END'
#!/bin/sh
-echo '1..0'
-exit 22
+echo 'Bail out!'
+exit 127
END
-chmod a+x one.test two.test
+cat > sighup.test << 'END'
+#!/bin/sh
+echo 'Bail out!'
+kill -1 $$
+END
+
+cat > sigterm.test << 'END'
+#!/bin/sh
+echo 'Bail out!'
+kill -15 $$
+END
+
+chmod a+x $tests
$MAKE check >stdout && { cat stdout; Exit 1; }
cat stdout
-# The 'prove' utility reports both the skip and the non-zero exit status,
-# so we do the same.
-count_test_results total=4 pass=0 fail=0 xpass=0 xfail=0 skip=2 error=2
-
-grep '^ERROR: one\.test - exited with status 1$' stdout
-grep '^ERROR: two\.test - exited with status 22$' stdout
+count_test_results total=4 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=4
+for tst in $tests; do grep "^ERROR: $tst - Bail out!" stdout; done
+$EGREP "ERROR: .*(exit.*status|terminat.*signal)" stdout && Exit 1
:
diff --git a/tests/tap-no-disable-hard-error.test
b/tests/tap-bailout-suppress-later-diagnostic.test
similarity index 67%
copy from tests/tap-no-disable-hard-error.test
copy to tests/tap-bailout-suppress-later-diagnostic.test
index 678e184..679343f 100755
--- a/tests/tap-no-disable-hard-error.test
+++ b/tests/tap-bailout-suppress-later-diagnostic.test
@@ -15,36 +15,31 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
# TAP support:
-# - "Bail out!" magic and TAP parse errors are not disabled nor turned
-# into simple failures by the definition DISABLE_HARD_ERRORS.
+# - A "Bail out!" directive causes the driver to ignore any TAP
+# diagnostic message in the rest of the following TAP stream.
parallel_tests=yes
. ./defs || Exit 1
-cat > Makefile.am << 'END'
-DISABLE_HARD_ERRORS = yes
-TEST_LOG_COMPILER = cat
-TESTS = bail.test few.test noplan.test
-END
-
. "$testsrcdir"/tap-setup.sh || fatal_ "sourcing tap-setup.sh"
-cat > bail.test <<END
-1..1
-Bail out!
-END
+echo AM_TEST_LOG_DRIVER_FLAGS = --comments >> Makefile
-cat > few.test <<END
+cat > all.test <<END
1..1
+# sanity check
+ok 1
+Bail out!
+# not seen
END
-cat > noplan.test <<END
-# nothing here
-END
+# Doing the sums above, we have:
$MAKE check >stdout && { cat stdout; Exit 1; }
cat stdout
-count_test_results total=3 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=3
+count_test_results total=2 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=1
+grep '# all.test: sanity check' stdout
+grep 'not seen' stdout && Exit 1
:
diff --git a/tests/tap-bailout-suppress-later-errors.test
b/tests/tap-bailout-suppress-later-errors.test
new file mode 100755
index 0000000..19aa9e2
--- /dev/null
+++ b/tests/tap-bailout-suppress-later-errors.test
@@ -0,0 +1,74 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - A "Bail out!" directive causes the driver to ignore any TAP
+# result or error in the rest of the following TAP stream.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+. "$testsrcdir"/tap-setup.sh || fatal_ "sourcing tap-setup.sh"
+
+# Various errors that can all be squashed into a single test script.
+cat > foo.test << 'END'
+1..5
+Bail out!
+# All possible test results.
+# Test out-of-order.
+ok 4
+# Extra TAP plan.
+1..2
+# Another bailout directive.
+Bail out! Not seen.
+# Stop now, with too few tests run.
+END
+
+# Tests run after a "SKIP" plan.
+cat > bar.test << 'END'
+1..0 # SKIP
+Bail out!
+ok 1
+END
+
+# Too many tests run.
+cat > baz.test << 'END'
+1..1
+ok 1
+Bail out!
+ok 2
+ok 3
+END
+
+TESTS='foo.test bar.test baz.test' $MAKE -e check >stdout \
+ && { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=5 pass=1 fail=0 xpass=0 xfail=0 skip=1 error=3
+
+grep '^ERROR: foo\.test - Bail out!$' stdout
+grep '^ERROR: bar\.test - Bail out!$' stdout
+grep '^SKIP: bar\.test' stdout
+grep '^ERROR: baz\.test - Bail out!$' stdout
+grep '^PASS: baz\.test 1$' stdout
+
+$FGREP 'Not seen' stdout && Exit 1
+
+test `$FGREP -c ': foo.test' stdout` -eq 1
+test `$FGREP -c ': bar.test' stdout` -eq 2
+test `$FGREP -c ': baz.test' stdout` -eq 2
+
+:
diff --git a/tests/tap-color.test b/tests/tap-color.test
index cd0329b..16fea15 100755
--- a/tests/tap-color.test
+++ b/tests/tap-color.test
@@ -44,7 +44,7 @@ AUTOMAKE_OPTIONS = color-tests
AM_TEST_LOG_DRIVER_FLAGS = --comments
TEST_LOG_COMPILER = cat
TESTS = all.test skip.test bail.test badplan.test noplan.test \
- few.test many.test order.test
+ few.test many.test order.test afterlate.test
END
. "$testsrcdir"/tap-setup.sh || fatal_ "sourcing tap-setup.sh"
@@ -96,6 +96,12 @@ cat > order.test << 'END'
ok 5
END
+cat > afterlate.test << 'END'
+ok 1
+1..2
+ok 2
+END
+
AM_COLOR_TESTS=always $MAKE check >stdout && { cat stdout; Exit 1; }
cat stdout
@@ -111,12 +117,12 @@ test_color ()
cat stdout | grep "^${grn}PASS${std}: bail\.test 1$"
cat stdout | grep "^${mgn}ERROR${std}: bail\.test - Bail out!$"
cat stdout | grep "^${mgn}ERROR${std}: badplan\.test - multiple test plans$"
- cat stdout | grep "^${mgn}ERROR${std}: badplan\.test 2 # AFTER LATE PLAN$"
cat stdout | grep "^${mgn}ERROR${std}: noplan\.test - missing test plan$"
cat stdout | grep "^${mgn}ERROR${std}: few.test - too few tests run
(expected 2, got 1)$"
cat stdout | grep "^${mgn}ERROR${std}: many.test - too many tests run
(expected 1, got 2)$"
cat stdout | grep "^${mgn}ERROR${std}: many.test 2 # UNPLANNED$"
cat stdout | grep "^${mgn}ERROR${std}: order.test 5 # OUT-OF-ORDER
(expecting 1)$"
+ cat stdout | grep "^${mgn}ERROR${std}: afterlate\.test 2 # AFTER LATE PLAN$"
# Diagnostic messages shouldn't be colorized.
cat stdout | grep "^# all\.test: Hi! I shouldn't be colorized!$"
:
diff --git a/tests/tap-global-result.test b/tests/tap-global-result.test
index 32e2401..7b83e6a 100755
--- a/tests/tap-global-result.test
+++ b/tests/tap-global-result.test
@@ -39,7 +39,8 @@ END
cat > skipall.test <<'END'
1..0 # SKIP
-not ok 1
+foo
+# bar
END
cat > fail.test <<'END'
@@ -56,8 +57,70 @@ END
(sed '1s/.*/1..4/' ok.test && echo 'ok 4 # TODO') > xpass2.test
-echo 'Bail out!' > error.test
-(cat ok.test && echo 'Bail out!') > error2.test
+echo 'Bail out!' > bail.test
+
+(cat ok.test && echo 'Bail out!') > bail2.test
+
+cat > bail3.test <<'END'
+1..0 # SKIP
+Bail out!
+END
+
+# Too many tests.
+cat > error.test <<'END'
+1..2
+ok 1
+ok 2 # SKIP
+not ok 3
+not ok 4 # TODO
+END
+
+# Too few tests.
+cat > error2.test <<'END'
+1..4
+ok 1
+not ok 2 # TODO
+ok 3 # SKIP
+END
+
+# Repeated plan.
+cat > error3.test <<'END'
+1..2
+1..2
+ok 1
+ok 2
+END
+
+# Too many tests, after a "SKIP" plan.
+cat > error4.test <<'END'
+1..0 # SKIP
+ok 1
+ok 2
+END
+
+# Tests out of order.
+cat > error5.test <<'END'
+1..4
+not ok 1 # TODO
+ok 3
+ok 2
+ok 4
+END
+
+# Wrong test number.
+cat > error6.test <<'END'
+1..2
+ok 1 # SKIP
+ok 7
+END
+
+# No plan.
+cat > error7.test <<'END'
+ok 1 # SKIP
+ok 2 # TODO
+not ok 3 # TODO
+ok 4
+END
cat > hodgepodge.test <<'END'
1..2
@@ -76,17 +139,13 @@ ok 4 # TODO
Bail out!
END
-# TODO: add scripts with TAP errors (multiple plans, out-of-order
-# tests, etc).
-
tests=`echo *.test` # Also required later.
TESTS="$tests" $MAKE -e check >stdout && { cat stdout; Exit 1; }
cat stdout
-# Dirty tricks required here.
-for tst in ok skip skipall fail fail2 xpass xpass2 error error2 \
- hodgepodge hodgepodge-all; do
+# Dirty trick required here.
+for tst in `echo " $tests " | sed 's/.test / /'`; do
echo :copy-in-global-log: yes >> $tst.trs
done
@@ -112,8 +171,16 @@ have_rst_section 'FAIL: fail'
have_rst_section 'FAIL: fail2'
have_rst_section 'FAIL: xpass'
have_rst_section 'FAIL: xpass2'
+have_rst_section 'ERROR: bail'
+have_rst_section 'ERROR: bail2'
+have_rst_section 'ERROR: bail3'
have_rst_section 'ERROR: error'
have_rst_section 'ERROR: error2'
+have_rst_section 'ERROR: error3'
+have_rst_section 'ERROR: error4'
+have_rst_section 'ERROR: error5'
+have_rst_section 'ERROR: error6'
+have_rst_section 'ERROR: error7'
have_rst_section 'ERROR: hodgepodge'
have_rst_section 'ERROR: hodgepodge-all'
diff --git a/tests/tap-plan-corner.test b/tests/tap-plan-corner.test
index eaebcf2..a818344 100755
--- a/tests/tap-plan-corner.test
+++ b/tests/tap-plan-corner.test
@@ -22,6 +22,8 @@ parallel_tests=yes
. "$testsrcdir"/tap-setup.sh || fatal_ "sourcing tap-setup.sh"
+# -------------------------------------------------------------------------
+
cat > leading-repeated.test <<END
1..1
1..1
@@ -42,6 +44,8 @@ for pos in leading trailing; do
grep "^ERROR: $pos-repeated\\.test - multiple test plans$" stdout
done
+# -------------------------------------------------------------------------
+
cat > leading-repeated.test <<END
1..2
ok 1
@@ -59,12 +63,44 @@ END
env TESTS="leading-repeated.test trailing-repeated.test" \
$MAKE -e check >stdout && { cat stdout; Exit 1; }
cat stdout
-count_test_results total=6 pass=2 fail=0 xpass=0 xfail=0 skip=0 error=4
-for pos in leading trailing; do
- grep "^ERROR: $pos-repeated\\.test - multiple test plans$" stdout
- grep "^ERROR: $pos-repeated\\.test 2 # AFTER LATE PLAN$" stdout
+count_test_results total=6 pass=3 fail=0 xpass=0 xfail=0 skip=0 error=3
+grep "^ERROR: leading-repeated\\.test - multiple test plans$" stdout
+grep "^ERROR: trailing-repeated\\.test - multiple test plans$" stdout
+grep "^ERROR: trailing-repeated\\.test 2 # AFTER LATE PLAN$" stdout
+grep "leading .*AFTER LATE PLAN" stdout && Exit 1
+
+# -------------------------------------------------------------------------
+
+cat > 1.test <<END
+1..0
+1..0
+END
+
+cat > 2.test <<END
+1..0 # SKIP
+1..0
+END
+
+cat > 3.test <<END
+1..0
+1..0 # SKIP
+END
+
+cat > 4.test <<END
+1..0 # SKIP
+1..0 # SKIP
+END
+
+env TESTS="1.test 2.test 3.test 4.test" \
+ $MAKE -e check >stdout && { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=8 pass=0 fail=0 xpass=0 xfail=0 skip=4 error=4
+for i in 1 2 3 4; do
+ grep "^ERROR: $i\\.test - multiple test plans$" stdout
done
+# -------------------------------------------------------------------------
+
cat > all.test <<END
1..5
ok 1
@@ -79,17 +115,17 @@ END
$MAKE -e check >stdout && { cat stdout; Exit 1; }
cat stdout
-count_test_results total=8 pass=2 fail=0 xpass=0 xfail=0 skip=0 error=6
+count_test_results total=8 pass=5 fail=0 xpass=0 xfail=0 skip=0 error=3
cat > exp <<'END'
PASS: all.test 1
PASS: all.test 2
ERROR: all.test - multiple test plans
-ERROR: all.test 3 # AFTER LATE PLAN
+PASS: all.test 3
ERROR: all.test - multiple test plans
-ERROR: all.test 4 # AFTER LATE PLAN
+PASS: all.test 4
ERROR: all.test - multiple test plans
-ERROR: all.test 5 # AFTER LATE PLAN
+PASS: all.test 5
END
$FGREP ': all.test' stdout > got
@@ -98,17 +134,55 @@ cat exp
cat got
diff exp got
-sed -e 1d all.test > t
-mv -f t all.test
+# -------------------------------------------------------------------------
+
+cat > all.test <<END
+1..2
+ok 1
+ok 2
+1..3
+ok 3
+END
+
+$MAKE -e check >stdout && { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=5 pass=2 fail=0 xpass=0 xfail=0 skip=0 error=3
+
+cat > exp <<'END'
+PASS: all.test 1
+PASS: all.test 2
+ERROR: all.test - multiple test plans
+ERROR: all.test 3 # UNPLANNED
+ERROR: all.test - too many tests run (expected 2, got 3)
+END
+
+$FGREP ': all.test' stdout > got
+
+cat exp
+cat got
+diff exp got
+
+# -------------------------------------------------------------------------
+
+cat > all.test <<END
+ok 1
+ok 2
+ok 3
+1..5
+1..5
+ok 4
+1..5
+ok 5
+END
$MAKE -e check >stdout && { cat stdout; Exit 1; }
cat stdout
-count_test_results total=7 pass=2 fail=0 xpass=0 xfail=0 skip=0 error=5
+count_test_results total=7 pass=3 fail=0 xpass=0 xfail=0 skip=0 error=4
cat > exp <<'END'
PASS: all.test 1
PASS: all.test 2
-ERROR: all.test 3 # AFTER LATE PLAN
+PASS: all.test 3
ERROR: all.test - multiple test plans
ERROR: all.test 4 # AFTER LATE PLAN
ERROR: all.test - multiple test plans
diff --git a/tests/tap-plan-errors.test b/tests/tap-plan-errors.test
index 95cc640..9d8abcd 100755
--- a/tests/tap-plan-errors.test
+++ b/tests/tap-plan-errors.test
@@ -19,9 +19,9 @@
# - multiple test plans
# - missing test plan
# - misplaced test plan (tests run after a late plan)
-# Checks about unplanned tests are performed in 'tap-unplanned.test'.
-# More checks about corner-cases in TAP plans are performed in
-# 'tap-plan-corner.test' and 'tap-plan-corner2.test'.
+# Checks about unplanned tests are performed in 'tap-unplanned.test'
+# and 'tap-skip-whole-badcount.test'. More checks about corner-cases
+# in TAP plans are performed in 'tap-plan-corner.test'.
parallel_tests=yes
. ./defs || Exit 1
@@ -76,6 +76,12 @@ ok 4
ok 5
END
+err='- missing test plan'
+my_check total=2 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+ok 1
+END
+
+
# The two test plans here are deliberately equal.
err='- multiple test plans'
my_check total=3 pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
@@ -85,9 +91,24 @@ ok 2
1..2
END
-err='- missing test plan'
-my_check total=2 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+# The second plan is diagnosed as extra, and only the first one is
+# relevant w.r.t. the number of the expected test.
+err='- multiple test plans'
+my_check total=4 pass=3 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+1..3
ok 1
+ok 2
+1..2
+ok 3
+END
+
+# As above, in a slightly different situation.
+err='- multiple test plans'
+my_check total=3 pass=2 fail=0 xpass=0 xfail=0 skip=0 error=1 <<END
+1..2
+ok 1
+ok 2
+1..4
END
:
diff --git a/tests/tap-planskip-and-logging.test
b/tests/tap-planskip-and-logging.test
new file mode 100755
index 0000000..621608b
--- /dev/null
+++ b/tests/tap-planskip-and-logging.test
@@ -0,0 +1,74 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - interactions between "TAP plan with SKIP" and logging of earlier or
+# later TAP or non-TAP text
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+. "$testsrcdir"/tap-setup.sh || fatal_ "sourcing tap-setup.sh"
+
+echo TEST_LOG_DRIVER_FLAGS = --comments >> Makefile
+
+cat > foo.test <<END
+1..0
+a non-TAP line
+# a comment
+END
+
+cat > foo2.test <<END
+a non-TAP line 2
+# a comment 2
+1..0
+END
+
+cat > bar.test <<END
+# an early comment
+an early non-TAP line
+ $tab
+1..0 # SKIP
+# a later comment
+a later non-TAP line
+END
+
+TESTS='foo.test foo2.test bar.test' $MAKE -e check >stdout \
+ || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=3 pass=0 fail=0 xpass=0 xfail=0 skip=3 error=0
+
+grep '^# foo\.test: a comment$' stdout
+grep '^# foo2\.test: a comment 2$' stdout
+grep '^# bar\.test: an early comment$' stdout
+grep '^# bar\.test: a later comment$' stdout
+
+cat foo.log
+cat foo2.log
+cat bar.log
+
+grep '^a non-TAP line$' foo.log
+grep '^# a comment$' foo.log
+grep '^a non-TAP line 2$' foo2.log
+grep '^# a comment 2$' foo2.log
+grep '^# an early comment' bar.log
+grep '^an early non-TAP line$' bar.log
+grep '^# a later comment' bar.log
+grep '^a later non-TAP line$' bar.log
+grep "^ $tab$" bar.log
+
+:
diff --git a/tests/tap-skip-whole-badexit.test b/tests/tap-planskip-badexit.test
similarity index 100%
rename from tests/tap-skip-whole-badexit.test
rename to tests/tap-planskip-badexit.test
diff --git a/tests/tap-not-ok-skip.test b/tests/tap-planskip-bailout.test
similarity index 75%
copy from tests/tap-not-ok-skip.test
copy to tests/tap-planskip-bailout.test
index f154d23..0048e00 100755
--- a/tests/tap-not-ok-skip.test
+++ b/tests/tap-planskip-bailout.test
@@ -15,25 +15,22 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
# TAP support:
-# - a "not ok # SKIP" line should count as a failure, for consistency
-# with the prove(1) utility.
+# - a "Bail out!" directive causes an hard error, even if coming after
+# a "SKIP plan" (e.g., "1..0 # SKIP").
parallel_tests=yes
. ./defs || Exit 1
. "$testsrcdir"/tap-setup.sh || fatal_ "sourcing tap-setup.sh"
-cat > all.test <<'END'
-1..4
-not ok # SKIP
-not ok 2 # SKIP
-not ok - foo # SKIP
-not ok 4 - bar # SKIP
+cat > all.test <<END
+1..0 # SKIP
+Bail out!
END
$MAKE check >stdout && { cat stdout; Exit 1; }
cat stdout
-
-count_test_results total=4 pass=0 fail=4 skip=0 xpass=0 xfail=0 error=0
+count_test_results total=2 pass=0 fail=0 xpass=0 xfail=0 skip=1 error=1
+grep '^ERROR: all\.test - Bail out!' stdout
:
diff --git a/tests/tap-no-spurious-summary.test b/tests/tap-planskip-late.test
similarity index 59%
copy from tests/tap-no-spurious-summary.test
copy to tests/tap-planskip-late.test
index bc7fd51..c2cb0a8 100755
--- a/tests/tap-no-spurious-summary.test
+++ b/tests/tap-planskip-late.test
@@ -15,33 +15,30 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
# TAP support:
-# - `:test-results:' directives in test scripts' output doesn't
-# originate spurious results in the testsuite summary
+# - the special "plan with SKIP" can also be used "late" in the TAP
+# stream, i.e., preceded by non-TAP output or TAP diagnostic.
parallel_tests=yes
. ./defs || Exit 1
. "$testsrcdir"/tap-setup.sh || fatal_ "sourcing tap-setup.sh"
-cat > all.test <<'END'
-1..1
-:test-result: PASS
-:test-result: FAIL
-:test-result: XPASS
-:test-result: XFAIL
-:test-result: SKIP
-:test-result: ERROR
-:test-result: UNKNOWN
-ok 1
+cat > foo.test <<END
+some non-TAP text, will be copied in the global log
+1..0 # SKIP from the last line
END
-$MAKE check >stdout || { cat stdout; Exit 1; }
-cat stdout
+cat > bar.test <<END
+# some TAP diagnostic, will go to console
+1..0
+END
-count_test_results total=1 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=0
+TESTS='foo.test bar.test' $MAKE -e check >stdout || { cat stdout; Exit 1; }
+cat stdout
-for result in PASS FAIL XPASS XFAIL SKIP ERROR UNKNOWN; do
- grep "^ *:test-result: $result$" all.log
-done
+grep '^SKIP: foo\.test .* from the last line$' stdout
+grep '^SKIP: bar\.test$' stdout
+test `grep -c ': .*\.test' stdout` -eq 2
+count_test_results total=2 pass=0 fail=0 xpass=0 xfail=0 skip=2 error=0
:
diff --git a/tests/tap-not-ok-skip.test b/tests/tap-planskip-later-errors.test
similarity index 75%
copy from tests/tap-not-ok-skip.test
copy to tests/tap-planskip-later-errors.test
index f154d23..eb2ee26 100755
--- a/tests/tap-not-ok-skip.test
+++ b/tests/tap-planskip-later-errors.test
@@ -15,25 +15,26 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
# TAP support:
-# - a "not ok # SKIP" line should count as a failure, for consistency
-# with the prove(1) utility.
+# - TAP errors following a "TAP plan with SKIP" are still diagnosed.
parallel_tests=yes
. ./defs || Exit 1
. "$testsrcdir"/tap-setup.sh || fatal_ "sourcing tap-setup.sh"
-cat > all.test <<'END'
-1..4
-not ok # SKIP
-not ok 2 # SKIP
-not ok - foo # SKIP
-not ok 4 - bar # SKIP
+cat > all.test <<END
+1..0
+a non-TAP line
+1..3
+1..5
+# a comment
+1..1
END
$MAKE check >stdout && { cat stdout; Exit 1; }
cat stdout
-count_test_results total=4 pass=0 fail=4 skip=0 xpass=0 xfail=0 error=0
+count_test_results total=4 pass=0 fail=0 xpass=0 xfail=0 skip=1 error=3
+test `grep -c '^ERROR: all\.test - multiple test plans' stdout` -eq 3
:
diff --git a/tests/tap-no-disable-hard-error.test
b/tests/tap-planskip-unplanned-corner.test
similarity index 56%
copy from tests/tap-no-disable-hard-error.test
copy to tests/tap-planskip-unplanned-corner.test
index 678e184..cf9f49e 100755
--- a/tests/tap-no-disable-hard-error.test
+++ b/tests/tap-planskip-unplanned-corner.test
@@ -15,36 +15,33 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
# TAP support:
-# - "Bail out!" magic and TAP parse errors are not disabled nor turned
-# into simple failures by the definition DISABLE_HARD_ERRORS.
+# - test results seen in a TAP stream that has a "plan with SKIP" are
+# flagged as errors, even if all of them are "SKIP" results.
parallel_tests=yes
. ./defs || Exit 1
-cat > Makefile.am << 'END'
-DISABLE_HARD_ERRORS = yes
-TEST_LOG_COMPILER = cat
-TESTS = bail.test few.test noplan.test
-END
-
. "$testsrcdir"/tap-setup.sh || fatal_ "sourcing tap-setup.sh"
-cat > bail.test <<END
-1..1
-Bail out!
+cat > foo.test <<END
+1..0 # SKIP
+ok 1 # SKIP
END
-cat > few.test <<END
-1..1
+cat > bar.test <<END
+ok 1 # SKIP
+1..0 # SKIP
END
-cat > noplan.test <<END
-# nothing here
-END
-
-$MAKE check >stdout && { cat stdout; Exit 1; }
+TESTS='foo.test bar.test' $MAKE -e check >stdout && { cat stdout; Exit 1; }
cat stdout
-count_test_results total=3 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=3
+count_test_results total=5 pass=0 fail=0 xpass=0 xfail=0 skip=2 error=3
+
+grep '^SKIP: foo\.test$' stdout
+grep '^ERROR: foo\.test 1 # UNPLANNED$' stdout
+grep '^ERROR: foo\.test - too many tests run (expected 0, got 1)$' stdout
+grep '^SKIP: bar\.test 1 # SKIP$' stdout
+grep '^ERROR: bar\.test - too many tests run (expected 0, got 1)$' stdout
:
diff --git a/tests/tap-planskip-unplanned.test
b/tests/tap-planskip-unplanned.test
new file mode 100755
index 0000000..6286dac
--- /dev/null
+++ b/tests/tap-planskip-unplanned.test
@@ -0,0 +1,73 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - a "plan with skip" given after one or more test result have already
+# been seen is an error
+# - any test result following a "plan with skip" is an error.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+. "$testsrcdir"/tap-setup.sh || fatal_ "sourcing tap-setup.sh"
+
+cat > all.test <<END
+ok 1
+1..0 # SKIP too late
+END
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=2 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=1
+grep '^ERROR: all\.test - too many tests run (expected 0, got 1)$' stdout
+
+cat > all.test <<END
+ok 1
+ok 2 # SKIP
+not ok 3 # TODO
+1..0 # SKIP too late
+END
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=4 pass=1 fail=0 xpass=0 xfail=1 skip=1 error=1
+grep '^ERROR: all\.test - too many tests run (expected 0, got 3)$' stdout
+
+cat > all.test <<END
+1..0 # SKIP falsified later
+ok 1
+END
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=3 pass=0 fail=0 xpass=0 xfail=0 skip=1 error=2
+grep '^ERROR: all\.test 1 # UNPLANNED$' stdout
+grep '^ERROR: all\.test - too many tests run (expected 0, got 1)$' stdout
+
+cat > all.test <<END
+1..0 # SKIP falsified later
+ok 1
+ok 2 # SKIP
+not ok 3
+not ok 4 # TODO
+END
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=6 pass=0 fail=0 xpass=0 xfail=0 skip=1 error=5
+grep '^ERROR: all\.test 1 # UNPLANNED$' stdout
+grep '^ERROR: all\.test 2 # UNPLANNED$' stdout
+grep '^ERROR: all\.test 3 # UNPLANNED$' stdout
+grep '^ERROR: all\.test 4 # UNPLANNED$' stdout
+grep '^ERROR: all\.test - too many tests run (expected 0, got 4)$' stdout
+
+:
diff --git a/tests/tap-skip-whole-whitespace.test
b/tests/tap-planskip-whitespace.test
similarity index 100%
rename from tests/tap-skip-whole-whitespace.test
rename to tests/tap-planskip-whitespace.test
diff --git a/tests/tap-skip-whole.test b/tests/tap-planskip.test
similarity index 89%
rename from tests/tap-skip-whole.test
rename to tests/tap-planskip.test
index 6b14be4..2852305 100755
--- a/tests/tap-skip-whole.test
+++ b/tests/tap-planskip.test
@@ -26,44 +26,35 @@ weirdchars=\''"$!&()[]<>#;^?*/@%=,.:'
cat > foo.test <<END
1..0
-not ok 1
-not ok 2
END
cat > bar.test <<END
+blah
+# blah
1..0$tab $tab
-ok 1
-Bail out! some random failure
END
# It is undefined whether the comment after the plan below should
# count as an explanation; still, the test should be skipped.
cat > baz.test <<END
1..0 # WWW::Mechanize not installed
-ok 1
+other
+ junk
+ lines
END
cat > wget.test <<END
1..0 # SKIP wget(1) not installed
-not ok 1 # TODO
-ok 2
-ok 3 # SKIP
+# See also curl.test
END
cat > curl.test <<END
1..0 # skip: Can't connect to gnu.org!
-not ok 1
-ok 2 # TODO
+# See also wget.test
END
cat > mu.test <<END
1..0 # Skip $weirdchars
-# Various TAP errors in here shouldn't be seen, as the test is skipped.
-1..2
-ok 1
-ok 22
-not ok 333
-1..7
END
env TESTS='foo.test bar.test baz.test wget.test curl.test mu.test' \
diff --git a/tests/tap-skip-whole-lastline.test
b/tests/tap-skip-whole-lastline.test
deleted file mode 100755
index 25697e0..0000000
--- a/tests/tap-skip-whole-lastline.test
+++ /dev/null
@@ -1,59 +0,0 @@
-#! /bin/sh
-# Copyright (C) 2011 Free Software Foundation, Inc.
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; either version 2, or (at your option)
-# any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program. If not, see <http://www.gnu.org/licenses/>.
-
-# TAP support:
-# - the special plan format to skip all the tests in a script can also
-# be used on the last line, even if preceded by non-TAP output, TAP
-# diagnostic, and even non-failing TAP test results.
-
-parallel_tests=yes
-. ./defs || Exit 1
-
-. "$testsrcdir"/tap-setup.sh || fatal_ "sourcing tap-setup.sh"
-
-echo AM_TEST_LOG_DRIVER_FLAGS = --comments >> Makefile
-
-cat > foo.test <<END
-some non-TAP text, will be copied in the global log
-1..0 # SKIP from the last line
-END
-
-cat > bar.test <<END
-# some TAP diagnostic, will go to console
-1..0
-END
-
-cat > baz.test <<END
-ok 1 - seen also if the whole test is skipped!
-1..0 # SKIP
-END
-
-st=0
-TESTS='foo.test bar.test baz.test' $MAKE -e check >stdout || st=$?
-cat stdout
-cat test-suite.log
-test $st -eq 0
-
-grep '^SKIP: foo\.test .* from the last line$' stdout
-grep '^SKIP: bar\.test$' stdout
-grep '^SKIP: baz\.test$' stdout
-grep '^# bar\.test: some TAP diagnostic, will go to console$' stdout
-grep '^some non-TAP text, will be copied in the global log$' test-suite.log
-grep '^PASS: baz\.test 1 - seen also if the whole test is skipped!$' stdout
-test `grep -c ': .*\.test' stdout` -eq 4
-count_test_results total=4 pass=1 fail=0 xpass=0 xfail=0 skip=3 error=0
-
-:
hooks/post-receive
--
GNU Automake
| [Prev in Thread] |
Current Thread |
[Next in Thread] |
- [Automake-commit] [SCM] GNU Automake branch, test-protocols, updated. v1.11-1017-g9e59677,
Stefano Lattarini <=