[FYI] {testsuite-work} testsuite: various fixlets and tweakings

[Stefano Lattarini]

* tests/seenc.test: Make grepping of automake stderr stricter.
Add a trailing `:' command.
* tests/symlink.test: Likewise.  Also, define `$am_create_testdir'
to "empty" to avoid bringing in unused auxiliary files (only to
have to remove them right away), and use proper m4 quoting in
`configure.in'.
* tests/vpath.test: Make grepping of generated `Makefile.in'
slightly stricter.  Prefer trailing `:' over trailing `Exit 0'.
* tests/suffix6c.test: Unset OBJEXT to avoid interferences from
the environment.
* tests/suffix12.test: Do not whitespace-indent `##' comments
when they are embedded in a makefile rule: having them indented
is not part of the Automake API, and might cause failures with
e.g., Tru64 make.
* tests/syntax.test: Simplify the `Makefile.am' to ensure that
automake doesn't fail for the wrong reasons.  Make grepping of
automake stderr slightly stricter.
* tests/test-harness-vpath-rewrite.test: Remove useless variable
definition from `Makefile.am'.
* tests/test-driver-custom-multitest.test: Fix typo in comments.
* tests/self-check-me.tap: Fix minor typo in test description.
* tests/vars3.test: Make grepping of automake stderr stricter
and safer.
* tests/version6.test: Add sanity check, verifying that the
version number extracted from `automake --version' output seems
legit.
* tests/auxdir2.test: Renamed ...
* tests/auxdir-compauted.tap: ... to this, and converted to the
use of TAP.
* tests/auxdir4.test: Renamed ...
* tests/auxdir-unportable.tap: ... to this, and converted to the
use of TAP.
* tests/auxdir3.test: Renamed ...
* tests/auxdir-misplaced.test: ... to this.
* tests/auxdir5.test: Renamed ...
* tests/auxdir-nonexistent.test: ... to this.
* tests/auxdir9.test: Renamed ...
* tests/auxdir-autodetect.test: ... to this.
* tests/Makefile.am (TESTS): Update.
(XFAIL_TESTS): Remove `auxdir2.test'.
---
 ChangeLog                                       |   44 +++++++++++++++++++++++
 tests/Makefile.am                               |   11 +++---
 tests/Makefile.in                               |   11 +++---
 tests/{auxdir9.test => auxdir-autodetect.test}  |    0
 tests/{auxdir2.test => auxdir-computed.tap}     |    8 +++--
 tests/{auxdir3.test => auxdir-misplaced.test}   |    0
 tests/{auxdir5.test => auxdir-nonexistent.test} |    0
 tests/{auxdir4.test => auxdir-unportable.tap}   |   22 +++++++++---
 tests/seenc.test                                |    8 +++-
 tests/self-check-me.tap                         |    2 +-
 tests/suffix12.test                             |    2 +-
 tests/suffix6c.test                             |    2 +
 tests/symlink.test                              |   14 +++++---
 tests/syntax.test                               |    4 +-
 tests/test-driver-custom-multitest.test         |    2 +-
 tests/test-harness-vpath-rewrite.test           |    1 -
 tests/vars3.test                                |   38 +++++++-------------
 tests/version6.test                             |    8 ++++
 tests/vpath.test                                |    5 ++-
 19 files changed, 123 insertions(+), 59 deletions(-)
 rename tests/{auxdir9.test => auxdir-autodetect.test} (100%)
 rename tests/{auxdir2.test => auxdir-computed.tap} (86%)
 rename tests/{auxdir3.test => auxdir-misplaced.test} (100%)
 rename tests/{auxdir5.test => auxdir-nonexistent.test} (100%)
 rename tests/{auxdir4.test => auxdir-unportable.tap} (73%)

diff --git a/ChangeLog b/ChangeLog
index e1d86f0..fb6bc35 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,5 +1,49 @@
 2011-08-19  Stefano Lattarini  <address@hidden>
 
+       testsuite: various fixlets and tweakings
+       * tests/seenc.test: Make grepping of automake stderr stricter.
+       Add a trailing `:' command.
+       * tests/symlink.test: Likewise.  Also, define `$am_create_testdir'
+       to "empty" to avoid bringing in unused auxiliary files (only to
+       have to remove them right away), and use proper m4 quoting in
+       `configure.in'.
+       * tests/vpath.test: Make grepping of generated `Makefile.in'
+       slightly stricter.  Prefer trailing `:' over trailing `Exit 0'.
+       * tests/suffix6c.test: Unset OBJEXT to avoid interferences from
+       the environment.
+       * tests/suffix12.test: Do not whitespace-indent `##' comments
+       when they are embedded in a makefile rule: having them indented
+       is not part of the Automake API, and might cause failures with
+       e.g., Tru64 make.
+       * tests/syntax.test: Simplify the `Makefile.am' to ensure that
+       automake doesn't fail for the wrong reasons.  Make grepping of
+       automake stderr slightly stricter.
+       * tests/test-harness-vpath-rewrite.test: Remove useless variable
+       definition from `Makefile.am'.
+       * tests/test-driver-custom-multitest.test: Fix typo in comments.
+       * tests/self-check-me.tap: Fix minor typo in test description.
+       * tests/vars3.test: Make grepping of automake stderr stricter
+       and safer.
+       * tests/version6.test: Add sanity check, verifying that the
+       version number extracted from `automake --version' output seems
+       legit.
+       * tests/auxdir2.test: Renamed ...
+       * tests/auxdir-compauted.tap: ... to this, and converted to the
+       use of TAP.
+       * tests/auxdir4.test: Renamed ...
+       * tests/auxdir-unportable.tap: ... to this, and converted to the
+       use of TAP.
+       * tests/auxdir3.test: Renamed ...
+       * tests/auxdir-misplaced.test: ... to this.
+       * tests/auxdir5.test: Renamed ...
+       * tests/auxdir-nonexistent.test: ... to this.
+       * tests/auxdir9.test: Renamed ...
+       * tests/auxdir-autodetect.test: ... to this.
+       * tests/Makefile.am (TESTS): Update.
+       (XFAIL_TESTS): Remove `auxdir2.test'.
+
+2011-08-19  Stefano Lattarini  <address@hidden>
+
        testsuite: fix some redundant autotools calls in tests on TAP
        * tests/tap-deps.test: Do not call autotools and configure; there
        is no need to, since this script sources `tap-setup.sh'.
diff --git a/tests/Makefile.am b/tests/Makefile.am
index b03d6a6..337f8c6 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -40,7 +40,6 @@ EXTRA_DIST = ChangeLog-old
 
 XFAIL_TESTS = \
 all.test \
-auxdir2.test \
 cond17.test \
 gcj6.test \
 override-conditional-2.test \
@@ -192,14 +191,14 @@ autohdr4.test \
 autohdrdry.test \
 automake-cmdline.tap \
 auxdir.test \
-auxdir2.test \
-auxdir3.test \
-auxdir4.test \
-auxdir5.test \
 auxdir6.test \
 auxdir7.test \
 auxdir8.test \
-auxdir9.test \
+auxdir-autodetect.test \
+auxdir-computed.tap \
+auxdir-misplaced.test \
+auxdir-nonexistent.test \
+auxdir-unportable.tap \
 backcompat.test \
 backcompat2.test \
 backcompat3.test \
diff --git a/tests/Makefile.in b/tests/Makefile.in
index 1dfcffd..beada3f 100644
--- a/tests/Makefile.in
+++ b/tests/Makefile.in
@@ -309,7 +309,6 @@ EXTRA_DIST = ChangeLog-old gen-parallel-tests 
parallel-tests.sh \
        extract-testsuite-summary tap-setup.sh tap-summary-aux.sh
 XFAIL_TESTS = \
 all.test \
-auxdir2.test \
 cond17.test \
 gcj6.test \
 override-conditional-2.test \
@@ -435,14 +434,14 @@ autohdr4.test \
 autohdrdry.test \
 automake-cmdline.tap \
 auxdir.test \
-auxdir2.test \
-auxdir3.test \
-auxdir4.test \
-auxdir5.test \
 auxdir6.test \
 auxdir7.test \
 auxdir8.test \
-auxdir9.test \
+auxdir-autodetect.test \
+auxdir-computed.tap \
+auxdir-misplaced.test \
+auxdir-nonexistent.test \
+auxdir-unportable.tap \
 backcompat.test \
 backcompat2.test \
 backcompat3.test \
diff --git a/tests/auxdir9.test b/tests/auxdir-autodetect.test
similarity index 100%
rename from tests/auxdir9.test
rename to tests/auxdir-autodetect.test
diff --git a/tests/auxdir2.test b/tests/auxdir-computed.tap
similarity index 86%
rename from tests/auxdir2.test
rename to tests/auxdir-computed.tap
index c915a6b..007eebc 100755
--- a/tests/auxdir2.test
+++ b/tests/auxdir-computed.tap
@@ -19,6 +19,8 @@
 
 . ./defs || Exit 1
 
+plan_ 3
+
 cat > configure.in <<END
 AC_INIT([$me], [1.0])
 AC_CONFIG_AUX_DIR([\$foo])
@@ -28,10 +30,10 @@ END
 
 : > Makefile.am
 
-$ACLOCAL
+command_ok_ "aclocal" $ACLOCAL
 # Both these two invocations are meant.
 # They exercise both code paths concerning auxiliary files.
-$AUTOMAKE -a
-$AUTOMAKE
+command_ok_ "automake -a" -D TODO -r "long-standing limitation" $AUTOMAKE -a
+command_ok_ "automake"    -D TODO -r "long-standing limitation" $AUTOMAKE
 
 :
diff --git a/tests/auxdir3.test b/tests/auxdir-misplaced.test
similarity index 100%
rename from tests/auxdir3.test
rename to tests/auxdir-misplaced.test
diff --git a/tests/auxdir5.test b/tests/auxdir-nonexistent.test
similarity index 100%
rename from tests/auxdir5.test
rename to tests/auxdir-nonexistent.test
diff --git a/tests/auxdir4.test b/tests/auxdir-unportable.tap
similarity index 73%
rename from tests/auxdir4.test
rename to tests/auxdir-unportable.tap
index 09bde4a..2eba8be 100755
--- a/tests/auxdir4.test
+++ b/tests/auxdir-unportable.tap
@@ -18,6 +18,8 @@
 
 . ./defs || Exit 1
 
+plan_ 4
+
 cat >configure.in <<END
 AC_INIT([$me], [1.0])
 AC_CONFIG_AUX_DIR([aux])
@@ -27,13 +29,23 @@ END
 
 : > Makefile.am
 
-$ACLOCAL
-AUTOMAKE_fails
-grep '^configure\.in:2:.*aux.*W32' stderr
+am_warn_unportable_auxdir ()
+{
+  details=$1
+  AUTOMAKE_fails
+  command_ok_ \
+    "warn about $details unportable auxdir name" \
+    grep '^configure\.in:2:.*aux.*W32' stderr
+}
+
+$ACLOCAL || fatal_ "aclocal failed"
+
+am_warn_unportable_auxdir "non-existent"
 
 if mkdir aux; then
-  AUTOMAKE_fails
-  grep '^configure\.in:2:.*aux.*W32' stderr
+  am_warn_unportable_auxdir "existent"
+else
+  skip_row_ 2 -r "cannot create directory named 'aux'"
 fi
 
 :
diff --git a/tests/seenc.test b/tests/seenc.test
index ea51cf5..de8bcc4 100755
--- a/tests/seenc.test
+++ b/tests/seenc.test
@@ -36,5 +36,9 @@ END
 
 $ACLOCAL
 AUTOMAKE_fails
-$FGREP CC stderr
-$FGREP CXX stderr
+grep ' C source.*CC.* undefined' stderr
+grep 'define .*CC.* add .*AC_PROG_CC' stderr
+grep ' C++ source.*CXX.* undefined' stderr
+grep 'define .*CXX.* add .*AC_PROG_CXX' stderr
+
+:
diff --git a/tests/self-check-me.tap b/tests/self-check-me.tap
index d7f61d6..f681f06 100755
--- a/tests/self-check-me.tap
+++ b/tests/self-check-me.tap
@@ -51,7 +51,7 @@ command_ok_ "override of \$me before ./defs causes no error" \
 r='ok'
 printf '%s\n' "$s" | grep '^me=foo\.test$' || r='not ok'
 printf '%s\n' "$s" | grep 'me=bad'         && r='not ok'
-result_ "$r" "override of \$me before defs is honored"
+result_ "$r" "override of \$me before ./defs is honored"
 unset r
 
 # Overriding $me after sourcing ./defs-static should work.
diff --git a/tests/suffix12.test b/tests/suffix12.test
index f583e36..430a350 100755
--- a/tests/suffix12.test
+++ b/tests/suffix12.test
@@ -31,7 +31,7 @@ AUTOMAKE_OPTIONS = subdir-objects
 SUFFIXES = .baz .o
 # We fake here:
 .baz.o:
-       ## Account for VPATH issues on weaker make implementations.
+## Account for VPATH issues on weaker make implementations.
        cp `test -f '$<' || echo $(srcdir)/`$< $@
 
 bin_PROGRAMS = foo
diff --git a/tests/suffix6c.test b/tests/suffix6c.test
index 149717d..7d1d7b5 100755
--- a/tests/suffix6c.test
+++ b/tests/suffix6c.test
@@ -31,6 +31,8 @@ AC_SUBST([OBJEXT])
 AC_OUTPUT
 END
 
+unset OBJEXT || :
+
 cat > Makefile.am << 'END'
 SUFFIXES = .zoo .o .obj address@hidden@
 
diff --git a/tests/symlink.test b/tests/symlink.test
index 5ecd4ab..790c566 100755
--- a/tests/symlink.test
+++ b/tests/symlink.test
@@ -18,20 +18,24 @@
 # Make sure we get an error if symlink creation fails.
 # Reported by Joerg-Martin Schwarz.
 
+am_create_testdir=empty
 . ./defs || Exit 1
 
-rm -f install-sh
-rm -f mkinstalldirs
-
 : > Makefile.am
 
 cat > configure.in << 'END'
 AC_INIT([symlink], [1.0])
-AC_CONFIG_AUX_DIR(sub)
+AC_CONFIG_AUX_DIR([subdir])
 AM_INIT_AUTOMAKE
 AC_CONFIG_FILES([Makefile])
 END
 
 $ACLOCAL
 AUTOMAKE_fails --add-missing
-grep 'error while making link' stderr
+
+grep '^configure\.in:3: .*missing.*error while making link' stderr
+grep '^configure\.in:3: .*install-sh.*error while making link' stderr
+
+test `$FGREP -c 'error while making link' stderr` -eq 2
+
+:
diff --git a/tests/syntax.test b/tests/syntax.test
index ba9249e..7834136 100755
--- a/tests/syntax.test
+++ b/tests/syntax.test
@@ -22,11 +22,11 @@
 cat > Makefile.am << 'END'
 foo = q \
 
-lib_LTLIBRARIES = foo.la
+bar = z
 END
 
 $ACLOCAL
 AUTOMAKE_fails
-grep 'Makefile\.am:2:.*blank line' stderr
+grep '^Makefile\.am:2:.*blank line follow.* trailing backslash' stderr
 
 :
diff --git a/tests/test-driver-custom-multitest.test 
b/tests/test-driver-custom-multitest.test
index 4fb023c..da15597 100755
--- a/tests/test-driver-custom-multitest.test
+++ b/tests/test-driver-custom-multitest.test
@@ -149,7 +149,7 @@ for vpath in : false; do
 
   # Check that the content of, and only of, the test logs with at least
   # one failing test case has been copied into `test-suite.log'.  Note
-  # that test logs containing skipped or failed test cases are *not*
+  # that test logs containing skipped or xfailed test cases are *not*
   # copied into `test-suite.log' -- a behaviour that deliberately differs
   # from the one of the built-in Automake test drivers.
   grep '%%' test-suite.log # For debugging.
diff --git a/tests/test-harness-vpath-rewrite.test 
b/tests/test-harness-vpath-rewrite.test
index 57ff909..f19506c 100755
--- a/tests/test-harness-vpath-rewrite.test
+++ b/tests/test-harness-vpath-rewrite.test
@@ -26,7 +26,6 @@ END
 
 cat > Makefile.am << 'END'
 LOG_COMPILER = true
-empty =
 TESTS = x0 foo bar baz x1
 EXTRA_DIST = $(TESTS)
 END
diff --git a/tests/vars3.test b/tests/vars3.test
index 5b6ff23..29dd575 100755
--- a/tests/vars3.test
+++ b/tests/vars3.test
@@ -43,34 +43,22 @@ cat stderr >&2
 # Lines number are printed in error message.
 # Use them to make sure errors are diagnosed against the right lines.
 
-# No error expected for these lines.
-grep 1: stderr
-grep 2: stderr && Exit 1
-grep 3: stderr && Exit 1
-grep 4: stderr
-grep 5: stderr
-grep 6: stderr && Exit 1
-grep 7: stderr
-grep 8: stderr
-grep 9: stderr
-grep 10: stderr && Exit 1
-grep 11: stderr && Exit 1
-grep 12: stderr && Exit 1
+# No error expected apart from those on these lines.
+grep -v '^Makefile\.am:[145789]:' stderr | grep . && Exit 1
 
 # Now check some individual values.
-grep 'shell echo' stderr
+grep ':1:.*shell echo' stderr
 grep 'nextvariableisbad' stderr && Exit 1
-grep 'addsuffix' stderr
-grep 'bad boy' stderr
-grep 'ok' stderr && Exit 1
-grep 'three errors' stderr
-grep 'on this' stderr
-grep 'long line' stderr
-grep 'o u c h' stderr
-grep 'wildcard' stderr
-grep 'another Error' stderr
-grep 'thisis' stderr && Exit 1
-grep 'here' stderr && Exit 1
+grep ':4:.*addsuffix' stderr
+grep ':5:.*bad boy' stderr
+grep ':7:.*three errors' stderr
+grep ':7:.*on this' stderr
+grep ':7:.*long line' stderr
+grep ':8:.*o u c h' stderr
+grep ':8:.*wildcard' stderr
+grep ':9:.*another Error' stderr
+
+$EGREP 'ok|thisis|here' stderr && Exit 1
 
 # None of these errors be diagnosed with `-Wno-portability'.
 $AUTOMAKE -Wno-portability
diff --git a/tests/version6.test b/tests/version6.test
index ca87e36..a7b6ca3 100755
--- a/tests/version6.test
+++ b/tests/version6.test
@@ -20,9 +20,17 @@
 
 amver=`$AUTOMAKE --version | sed -e 's/.* //;1q'`
 
+# Does the extracted version number seems legit?
+case $amver in
+  *[0-9].[0-9]*) ;;
+  *) fatal_ "couldn't extract version number from automake" ;;
+esac
+
 cat > Makefile.am << END
 AUTOMAKE_OPTIONS = $amver
 END
 
 $ACLOCAL
 $AUTOMAKE
+
+:
diff --git a/tests/vpath.test b/tests/vpath.test
index 4cbd5a1..76d3a53 100755
--- a/tests/vpath.test
+++ b/tests/vpath.test
@@ -27,5 +27,8 @@ END
 $ACLOCAL
 $AUTOMAKE
 
+grep VPATH Makefile.in # For debugging.
+grep '^VPATH = zardoz$' Makefile.in
 grep 'address@hidden@' Makefile.in && Exit 1
-Exit 0
+
+:
-- 
1.7.2.3