[Automake-commit] [SCM] GNU Automake branch, test-protocols, updated. v1.11-1035-g40705c5

[Stefano Lattarini]

This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "GNU Automake".

http://git.sv.gnu.org/gitweb/?p=automake.git;a=commitdiff;h=40705c5cc5898ff924fea0c6f54f8953205a630d

The branch, test-protocols has been updated
       via  40705c5cc5898ff924fea0c6f54f8953205a630d (commit)
      from  271343d8ef023b9891d9b06bce5e62268f8b50e5 (commit)

Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.

- Log -----------------------------------------------------------------
commit 40705c5cc5898ff924fea0c6f54f8953205a630d
Author: Stefano Lattarini <address@hidden>
Date:   Fri Aug 19 16:47:12 2011 +0200

    coverage: some more tests on corner cases of TAP support
    
    * tests/tap-no-spurious.test: Extend checks verifying that a line
    matching, say, the regex "^ok[a-zA-Z0-9_]" is *not* considered a
    TAP result.
    * tests/tap-no-spurious-numbers.test: New test, checking that our
    TAP driver doesn't spuriously recognize as TAP result numbers what
    is not (even if it seems pretty close).
    * tests/tap-negative-numbers.test: New test, checking that our
    TAP driver doesn't spuriously recognize negative TAP result
    numbers, but correctly interprets them as test descriptions
    instead.
    * tests/tap-plan-leading-zero.test, tap-numbers-leading-zero.test:
    New tests, checking how our driver fares in recognizing numbers
    with leading zeroes in TAP results or TAP plans.
    * tests/tap-planskip-malformed.test: New test, checking that a
    malformed TAP plan is not recognized as a valid plan.
    * tests/tap-plan-whitespace.test: New test, checking that a TAP
    plan line with trailing whitespace is recognized and handled
    correctly.
    * tests/Makefile.am (tap_with_common_setup_tests): Update.

-----------------------------------------------------------------------

Summary of changes:
 ChangeLog                                          |   23 ++++
 tests/Makefile.am                                  |    6 +
 tests/Makefile.in                                  |    5 +
 ...-msg0-result.test => tap-negative-numbers.test} |   47 ++++-----
 tests/tap-no-spurious-numbers.test                 |   99 ++++++++++++++++++
 tests/tap-no-spurious.test                         |   25 ++++-
 tests/tap-numbers-leading-zero.test                |  109 ++++++++++++++++++++
 .../{tap-plan.test => tap-plan-leading-zero.test}  |   43 +++++----
 ...ble-hard-error.test => tap-plan-malformed.test} |   44 +++++---
 ...ty-diagnostic.test => tap-plan-whitespace.test} |   22 ++--
 10 files changed, 346 insertions(+), 77 deletions(-)
 copy tests/{tap-msg0-result.test => tap-negative-numbers.test} (63%)
 create mode 100755 tests/tap-no-spurious-numbers.test
 create mode 100755 tests/tap-numbers-leading-zero.test
 copy tests/{tap-plan.test => tap-plan-leading-zero.test} (54%)
 copy tests/{tap-no-disable-hard-error.test => tap-plan-malformed.test} (58%)
 copy tests/{tap-empty-diagnostic.test => tap-plan-whitespace.test} (72%)

diff --git a/ChangeLog b/ChangeLog
index f86d1cd..bdccc31 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,26 @@
+2011-08-20  Stefano Lattarini  <address@hidden>
+
+       coverage: some more tests on corner cases of TAP support
+       * tests/tap-no-spurious.test: Extend checks verifying that a line
+       matching, say, the regex "^ok[a-zA-Z0-9_]" is *not* considered a
+       TAP result.
+       * tests/tap-no-spurious-numbers.test: New test, checking that our
+       TAP driver doesn't spuriously recognize as TAP result numbers what
+       is not (even if it seems pretty close).
+       * tests/tap-negative-numbers.test: New test, checking that our
+       TAP driver doesn't spuriously recognize negative TAP result
+       numbers, but correctly interprets them as test descriptions
+       instead.
+       * tests/tap-plan-leading-zero.test, tap-numbers-leading-zero.test:
+       New tests, checking how our driver fares in recognizing numbers
+       with leading zeroes in TAP results or TAP plans.
+       * tests/tap-planskip-malformed.test: New test, checking that a
+       malformed TAP plan is not recognized as a valid plan.
+       * tests/tap-plan-whitespace.test: New test, checking that a TAP
+       plan line with trailing whitespace is recognized and handled
+       correctly.
+       * tests/Makefile.am (tap_with_common_setup_tests): Update.
+
 2011-08-19  Stefano Lattarini  <address@hidden>
 
        testsuite: fix some redundant autotools calls in tests on TAP
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 2d630a9..e7f8570 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -1171,9 +1171,12 @@ tap-merge-stdout-stderr.test \
 tap-no-merge-stdout-stderr.test \
 tap-no-disable-hard-error.test \
 tap-no-spurious-summary.test \
+tap-no-spurious-numbers.test \
 tap-no-spurious.test \
 tap-not-ok-skip.test \
 tap-numeric-description.test \
+tap-negative-numbers.test \
+tap-numbers-leading-zero.test \
 tap-out-of-order.test \
 tap-passthrough.test \
 tap-passthrough-exit.test \
@@ -1181,6 +1184,9 @@ tap-plan.test \
 tap-plan-corner.test \
 tap-plan-errors.test \
 tap-plan-middle.test \
+tap-plan-whitespace.test \
+tap-plan-leading-zero.test \
+tap-plan-malformed.test \
 tap-missing-plan-and-bad-exit.test \
 tap-planskip.test \
 tap-planskip-late.test \
diff --git a/tests/Makefile.in b/tests/Makefile.in
index dc7cb80..96edc94 100644
--- a/tests/Makefile.in
+++ b/tests/Makefile.in
@@ -1412,9 +1412,12 @@ tap-merge-stdout-stderr.test \
 tap-no-merge-stdout-stderr.test \
 tap-no-disable-hard-error.test \
 tap-no-spurious-summary.test \
+tap-no-spurious-numbers.test \
 tap-no-spurious.test \
 tap-not-ok-skip.test \
 tap-numeric-description.test \
+tap-negative-numbers.test \
+tap-numbers-leading-zero.test \
 tap-out-of-order.test \
 tap-passthrough.test \
 tap-passthrough-exit.test \
@@ -1422,6 +1425,7 @@ tap-plan.test \
 tap-plan-corner.test \
 tap-plan-errors.test \
 tap-plan-middle.test \
+tap-plan-leading-zero.test \
 tap-missing-plan-and-bad-exit.test \
 tap-planskip.test \
 tap-planskip-late.test \
@@ -1432,6 +1436,7 @@ tap-planskip-whitespace.test \
 tap-planskip-badexit.test \
 tap-planskip-bailout.test \
 tap-planskip-later-errors.test \
+tap-planskip-malformed.test \
 tap-realtime.test \
 tap-recheck-logs.test \
 tap-todo-skip-together.test \
diff --git a/tests/tap-msg0-result.test b/tests/tap-negative-numbers.test
similarity index 63%
copy from tests/tap-msg0-result.test
copy to tests/tap-negative-numbers.test
index 86259e5..9fdee87 100755
--- a/tests/tap-msg0-result.test
+++ b/tests/tap-negative-numbers.test
@@ -15,43 +15,38 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 # TAP support:
-#  - the string "0" as a test description
+#  - don't spuriously recognize negative TAP result numbers, but correctly
+#    interpret them as test descriptions instead
 
 parallel_tests=yes
 . ./defs || Exit 1
 
 . "$testsrcdir"/tap-setup.sh || fatal_ "sourcing tap-setup.sh"
 
-cat > all.test << 'END'
-1..10
-ok 1 0
-ok - 0
-not ok 3 0
-not ok - 0
-ok 5 0 # TODO
-ok - 0 # TODO
-not ok 7 0 # TODO
-not ok - 0 # TODO
-ok 9 0 # SKIP
-ok - 0 # SKIP
+cat > all.test <<'END'
+1..7
+ok -1
+not ok -3
+ok -2 # SKIP
+not ok -5 # TODO
+ok -04 # TODO
+ok -121
+not ok -50000
 END
 
 $MAKE check >stdout && { cat stdout; Exit 1; }
 cat stdout
 
-count_test_results total=10 pass=2 fail=2 xpass=2 xfail=2 skip=2 error=0
-
-cat > exp << 'END'
-PASS: all.test 1 0
-PASS: all.test 2 - 0
-FAIL: all.test 3 0
-FAIL: all.test 4 - 0
-XPASS: all.test 5 0 # TODO
-XPASS: all.test 6 - 0 # TODO
-XFAIL: all.test 7 0 # TODO
-XFAIL: all.test 8 - 0 # TODO
-SKIP: all.test 9 0 # SKIP
-SKIP: all.test 10 - 0 # SKIP
+count_test_results total=7 pass=2 fail=2 xpass=1 xfail=1 skip=1 error=0
+
+cat > exp <<'END'
+PASS: all.test 1 -1
+FAIL: all.test 2 -3
+SKIP: all.test 3 -2 # SKIP
+XFAIL: all.test 4 -5 # TODO
+XPASS: all.test 5 -04 # TODO
+PASS: all.test 6 -121
+FAIL: all.test 7 -50000
 END
 
 $FGREP ': all.test' stdout > got
diff --git a/tests/tap-no-spurious-numbers.test 
b/tests/tap-no-spurious-numbers.test
new file mode 100755
index 0000000..e3ee0c8
--- /dev/null
+++ b/tests/tap-no-spurious-numbers.test
@@ -0,0 +1,99 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+#  - we shouldn't spuriously recognize as TAP result numbers what it
+#    not, even if it seems pretty close
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+. "$testsrcdir"/tap-setup.sh || fatal_ "sourcing tap-setup.sh"
+
+cat > prefixes <<'END'
+A
+a
+_
++
+-
+=
+/
+*
+.
+:
+,
+;
+$
+@
+%
+&
+#
+?
+!
+|
+\
+"
+`
+'
+(
+)
+[
+]
+{
+}
+<
+>
+END
+
+n=`wc -l <prefixes`
+
+# See the loop below to understand this initialization.
+pass=`expr $n '*' 3`
+fail=$pass
+skip=`expr $pass - 3`
+xfail=$skip
+xpass=$xfail
+error=0
+total=`expr $pass + $fail + $skip + $xfail + $xpass`
+
+echo 1..$total > all.test
+
+highno=1000
+
+for result in 'ok' 'not ok'; do
+  for spacing in "" " " "$tab"; do
+    subst="$result &$spacing$higno"
+    sed -e "s|.*|$subst|" prefixes
+    for directive in TODO SKIP; do
+      test "$result $directive" != "not ok SKIP" || continue
+      sed -e '/^#$/d' -e "s|.*|$subst # $directive|" prefixes
+    done
+  done
+done >> all.test
+
+cat all.test # For debugging.
+
+# Sanity checks.
+grep '#.*#' all.test && framework_failure_ "creating all.test"
+test `wc -l <all.test` -lt $highno || framework_failure_ "creating all.test"
+
+$MAKE check >stdout || :
+cat stdout
+
+count_test_results total=$total pass=$pass fail=$fail skip=$skip \
+                   xpass=$xpass xfail=$xfail error=$error
+
+:
diff --git a/tests/tap-no-spurious.test b/tests/tap-no-spurious.test
index 11e2d1a..3e2afac 100755
--- a/tests/tap-no-spurious.test
+++ b/tests/tap-no-spurious.test
@@ -46,14 +46,10 @@ cat >> all.test <<END
  ok 1
 ${tab}ok
 ${tab}ok 1
-ok1
-ok23
  not ok
  not ok 1
 ${tab}not ok
 ${tab}not ok 1
-not ok1
-not ok23
 notok
 notok 1
 not${tab}ok
@@ -72,6 +68,20 @@ no ok 1
 #not ok 1
 END
 
+set +x # Don't pollute logs too much.
+for r in 'ok' 'not ok'; do
+  for s1 in \
+    a b c d e f g h i j k l m n o p q r s t u v w x y z \
+    A B C D E F G H I J K L M N O P Q R S T U V W X Y Z \
+    0 1 2 3 4 5 6 7 8 9 _ ab 0a 23 a1B2c _o _x_y_
+  do
+    for s2 in '' @ % + - = / . : \; \* \? \& \! \# \$ \< \> \\; do
+      printf '%s\n' "$r$s1$s2"
+    done
+  done
+done >> all.test
+set -x # Reset shell xtraces.
+
 # The prove(1) utility doesn't bail out on these, so our driver
 # shouldn't either.
 cat >> all.test <<'END'
@@ -88,6 +98,13 @@ Bail out
 # Bail out!
 END
 
+cat all.test # For debugging.
+
+# Minor sanity check.
+test `grep -c '^ok1$' all.test` -eq 1 \
+  && test `grep -c '^not ok1$' all.test` -eq 1 \
+  || framework_failure_ "creating all.test"
+
 $MAKE check >stdout || { cat stdout; Exit 1; }
 cat stdout
 
diff --git a/tests/tap-numbers-leading-zero.test 
b/tests/tap-numbers-leading-zero.test
new file mode 100755
index 0000000..81c9797
--- /dev/null
+++ b/tests/tap-numbers-leading-zero.test
@@ -0,0 +1,109 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+#  - how does TAP result numbers with leading zero fares?
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+. "$testsrcdir"/tap-setup.sh || fatal_ "sourcing tap-setup.sh"
+
+do_checks ()
+{
+  $MAKE check >stdout && { cat stdout; Exit 1; }
+  cat stdout
+  count_test_results "$@"
+  # Allow some normalization of leading zeros, without forcing it.
+  LC_ALL=C sed -n 's/: all\.test 0*\([0-9]\)/: all.test \1/p' stdout > got
+  cat exp
+  cat got
+  diff exp got
+}
+
+cat > all.test <<END
+1..15
+
+ok 01
+ok 0002
+
+not ok 03
+not ok 0004
+
+ok 05 # SKIP
+ok 0006 # SKIP
+
+not ok 07 # TODO
+not ok 0008 # TODO
+
+ok 009
+ok 010
+
+ok 00000011
+not ok 0012
+ok 00000013 # SKIP
+not ok 0014 # TODO
+ok 00000015 # TODO
+END
+
+cat > exp <<END
+PASS: all.test 1
+PASS: all.test 2
+FAIL: all.test 3
+FAIL: all.test 4
+SKIP: all.test 5 # SKIP
+SKIP: all.test 6 # SKIP
+XFAIL: all.test 7 # TODO
+XFAIL: all.test 8 # TODO
+PASS: all.test 9
+PASS: all.test 10
+PASS: all.test 11
+FAIL: all.test 12
+SKIP: all.test 13 # SKIP
+XFAIL: all.test 14 # TODO
+XPASS: all.test 15 # TODO
+END
+
+do_checks total=15 pass=5 fail=3 xpass=1 xfail=3 skip=3 error=0
+
+cat > all.test <<END
+1..8
+
+ok 010
+not ok 003
+ok 0001 # SKIP
+not ok 010 # TODO
+ok 00100 # TODO
+
+ok 06
+ok 00007
+ok 8
+END
+
+cat > exp <<END
+ERROR: all.test 10 # OUT-OF-ORDER (expecting 1)
+ERROR: all.test 3 # OUT-OF-ORDER (expecting 2)
+ERROR: all.test 1 # OUT-OF-ORDER (expecting 3)
+ERROR: all.test 10 # OUT-OF-ORDER (expecting 4)
+ERROR: all.test 100 # OUT-OF-ORDER (expecting 5)
+PASS: all.test 6
+PASS: all.test 7
+PASS: all.test 8
+END
+
+do_checks total=8 pass=3 fail=0 xpass=0 xfail=0 skip=0 error=5
+
+:
diff --git a/tests/tap-plan.test b/tests/tap-plan-leading-zero.test
similarity index 54%
copy from tests/tap-plan.test
copy to tests/tap-plan-leading-zero.test
index e2fda35..4c54649 100755
--- a/tests/tap-plan.test
+++ b/tests/tap-plan-leading-zero.test
@@ -15,38 +15,43 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 # TAP support:
-#  - test scripts with the test plan at the beginning
-#  - test scripts with the test plan at the end
+#  - TAP plans with numbers having leading zeroes, as in "1..01"
+#  - TAP "SKIP" plans with multiple zeroes, as in "1..00 # SKIP"
+# This is consistent with the behaviour of the `prove' utility.
 
 parallel_tests=yes
 . ./defs || Exit 1
 
 . "$testsrcdir"/tap-setup.sh || fatal_ "sourcing tap-setup.sh"
 
-cat > top.test <<END
-1..3
+cat > a.test <<END
+1..01
 ok 1
-ok 2 # SKIP
-ok 3
-# a trailing comment
 END
 
-cat > bot.test <<END
+cat > b.test <<END
+1..0002
 ok 1
-# a comment
 ok 2
-not ok 3 # TODO
-# another comment
-ok
-1..4
 END
 
-for tap_flags in "" "--comments"; do
-  env TEST_LOG_DRIVER_FLAGS="$tap_flags" TESTS='top.test bot.test' \
-    $MAKE -e check >stdout || { cat stdout; Exit 1; }
-  cat stdout
-  count_test_results total=7 pass=5 xfail=1 skip=1 fail=0 xpass=0 error=0
-done
+echo 1..010 > c.test
+for i in 1 2 3 4 5 6 7 8 9 10; do echo ok $i; done >> c.test
 
+echo 1..00100 > d.test
+for i in 1 2 3 4 5 6 7 8 9 10; do
+  for j in 1 2 3 4 5 6 7 8 9 10; do
+    echo ok
+  done
+done >> d.test
+
+echo 1..00 > e.test
+echo '1..000 # SKIP' > f.test
+
+env TESTS='a.test b.test c.test d.test e.test f.test' \
+  $MAKE -e check >stdout || { cat stdout; Exit 1; }
+cat stdout
+
+count_test_results total=115 pass=113 xfail=0 skip=2 fail=0 xpass=0 error=0
 
 :
diff --git a/tests/tap-no-disable-hard-error.test 
b/tests/tap-plan-malformed.test
similarity index 58%
copy from tests/tap-no-disable-hard-error.test
copy to tests/tap-plan-malformed.test
index 678e184..9eb8698 100755
--- a/tests/tap-no-disable-hard-error.test
+++ b/tests/tap-plan-malformed.test
@@ -14,37 +14,47 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-# TAP support:
-#  - "Bail out!" magic and TAP parse errors are not disabled nor turned
-#    into simple failures by the definition DISABLE_HARD_ERRORS.
+# TAP support: a malformed TAP plan is not recognized.  The checks in
+# here should be consistent with the behaviour of the `prove' utility.
 
 parallel_tests=yes
 . ./defs || Exit 1
 
-cat > Makefile.am << 'END'
-DISABLE_HARD_ERRORS = yes
-TEST_LOG_COMPILER = cat
-TESTS = bail.test few.test noplan.test
+. "$testsrcdir"/tap-setup.sh || fatal_ "sourcing tap-setup.sh"
+
+cat > a.test <<END
+1..1 foo
 END
 
-. "$testsrcdir"/tap-setup.sh || fatal_ "sourcing tap-setup.sh"
+cat > b.test <<END
+ 1..2
+ok 1
+ok 2
+END
 
-cat > bail.test <<END
-1..1
-Bail out!
+cat > c.test <<END
+1..1 # SKIP
 END
 
-cat > few.test <<END
-1..1
+cat > d.test <<END
+1..2 # foo bar
+not ok 1 # TODO
+not ok 2 # TODO
 END
 
-cat > noplan.test <<END
-# nothing here
+cat > e.test <<END
+0..0
 END
 
-$MAKE check >stdout && { cat stdout; Exit 1; }
+tests_list=`echo *.test`
+
+TESTS="$tests_list" $MAKE -e check >stdout && { cat stdout; Exit 1; }
 cat stdout
 
-count_test_results total=3 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=3
+count_test_results total=9 pass=2 fail=0 xpass=0 xfail=2 skip=0 error=5
+
+for tst in $tests_list; do
+  grep "^ERROR: $tst - missing test plan$" stdout
+done
 
 :
diff --git a/tests/tap-empty-diagnostic.test b/tests/tap-plan-whitespace.test
similarity index 72%
copy from tests/tap-empty-diagnostic.test
copy to tests/tap-plan-whitespace.test
index 6bcb8ce..3cdd79d 100755
--- a/tests/tap-empty-diagnostic.test
+++ b/tests/tap-plan-whitespace.test
@@ -15,27 +15,27 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 # TAP support:
-#  - empty diagnostic messages are discarder
+#  - plan line with trailing whitespace is recognized and handled correctly
 
 parallel_tests=yes
 . ./defs || Exit 1
 
 . "$testsrcdir"/tap-setup.sh || fatal_ "sourcing tap-setup.sh"
 
-sed 's/\$$//' > all.test <<END
-1..1$
+sed 's/\$$//' > foo.test <<END
+1..2 $
 ok 1$
-#$
-# $
-#${tab}$
-#   ${tab} $tab${tab}$
+ok 2$
 END
 
-$MAKE check >stdout || { cat stdout; Exit 1; }
+cat > bar.test <<END
+1..1$tab   $tab$tab
+ok 1
+END
+
+TESTS='foo.test bar.test' $MAKE -e check >stdout || { cat stdout; Exit 1; }
 cat stdout
 
-grep '^PASS:.*all\.test' stdout # Sanity check.
-grep '#.*all\.test' stdout && Exit 1
-grep "all\.test[ $tab]*:[ $tab]*$" stdout && Exit 1
+count_test_results total=3 pass=3 fail=0 error=0 xpass=0 xfail=0 skip=0
 
 :


hooks/post-receive
-- 
GNU Automake