[PATCH] {testsuite-work} tests: autogenerate list of wrapped tests for `lib/' shell scripts (was: Re: [PATCH 3/3] {testsuite-work} tests: `lib/' shell scripts transparently tested also with $SHELL)

[Stefano Lattarini]

On Tuesday 07 June 2011, Stefano Lattarini wrote:
> On Tuesday 07 June 2011, Peter Rosin wrote:
> > > +
> > > +# FIXME: this list might probably be made auto-generated, like
> > > +# $(parallel_tests) is...  But would that worth the hassle?
> > > +config_shell_tests = \
> > > +  compile-w.shtst \
> > > +  compile2-w.shtst \
> > > +  compile3-w.shtst \
> > > +  compile4-w.shtst \
> > > +  compile5-w.shtst \
> > > +  compile6-w.shtst \
> > > +  instsh2-w.shtst \
> > > +  instsh3-w.shtst \
> > > +  mkinst3-w.shtst \
> > > +  missing-w.shtst \
> > > +  missing2-w.shtst \
> > > +  missing3-w.shtst \
> > > +  missing5-w.shtst
> > 
> > I expected to see the ar-lib tests here? Or did you not add those
> > since the AM_PROG_AR macro hasn't been merged? But would it hurt
> > to test ar-lib with $SHELL prior to that merge?
> >
> No, this is just a stupid oversight on my part, sorry.  Which probably
> shows that we indeed should make this list automatically-generated
> after all.  But that's for a follow-up patch IMHO.
> 
> Thanks,
>   Stefano
> 
And here it is.  I'll push in 48 hours if there is no objection by then.

Regards,
  Stefano

From 5ac7081cc229a57df779a134de5b192c6d3519a7 Mon Sep 17 00:00:00 2001
Message-Id: <address@hidden>
From: Stefano Lattarini <address@hidden>
Date: Mon, 13 Jun 2011 14:48:57 +0200
Subject: [PATCH] tests: autogenerate list of wrapped tests for `lib/' shell 
scripts

* tests/gen-config-shell-tests: New script, generates distributed
makefile snippet `tests/config-shell-tests.am' to list all tests
that use the `get_shell_script' function, with names mangled to
use suffix `-w.shtst', in ...
* tests/Makefile.am (config_shell_tests): ... this macro, whose
definition has been consequently removed from Makefile.am.
(EXTRA_DIST): Distribute the new script.
($(srcdir)/config-shell-tests.am): Generate using the new script.
(include): Include the `config-shell-tests.am' fragment.
* bootstrap: Invoke `tests/gen-config-shell-tests' to generate
`tests/config-shell-tests.am'.
* tests/.gitignore: Ignore `config-shell-tests.am'.
* tests/gen-parallel-tests: Fixlet in heading comments.
---
 ChangeLog                    |   17 ++++++++++
 bootstrap                    |    1 +
 tests/.gitignore             |    1 +
 tests/Makefile.am            |   26 +++------------
 tests/Makefile.in            |   54 +++++++++++++++++---------------
 tests/gen-config-shell-tests |   70 ++++++++++++++++++++++++++++++++++++++++++
 tests/gen-parallel-tests     |    8 ++--
 7 files changed, 128 insertions(+), 49 deletions(-)
 create mode 100755 tests/gen-config-shell-tests

diff --git a/ChangeLog b/ChangeLog
index c202a6d..d8ca3bc 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,5 +1,22 @@
 2011-06-13  Stefano Lattarini  <address@hidden>
 
+       tests: autogenerate list of wrapped tests for `lib/' shell scripts
+       * tests/gen-config-shell-tests: New script, generates distributed
+       makefile snippet `tests/config-shell-tests.am' to list all tests
+       that use the `get_shell_script' function, with names mangled to
+       use suffix `-w.shtst', in ...
+       * tests/Makefile.am (config_shell_tests): ... this macro, whose
+       definition has been consequently removed from Makefile.am.
+       (EXTRA_DIST): Distribute the new script.
+       ($(srcdir)/config-shell-tests.am): Generate using the new script.
+       (include): Include the `config-shell-tests.am' fragment.
+       * bootstrap: Invoke `tests/gen-config-shell-tests' to generate
+       `tests/config-shell-tests.am'.
+       * tests/.gitignore: Ignore `config-shell-tests.am'.
+       * tests/gen-parallel-tests: Fixlet in heading comments.
+
+2011-06-13  Stefano Lattarini  <address@hidden>
+
        tests: test mdate-sh with /bin/sh too
        * tests/mdate5.test: Fetch the `mdate-sh' script using the
        `get_shell_script' function, and run it directly instead of
diff --git a/bootstrap b/bootstrap
index 95e51d1..aafc6eb 100755
--- a/bootstrap
+++ b/bootstrap
@@ -106,6 +106,7 @@ dosubst automake.in automake.tmp
 # Create required makefile snippets.
 cd tests
 $BOOTSTRAP_SHELL ./gen-parallel-tests > parallel-tests.am
+$BOOTSTRAP_SHELL ./gen-config-shell-tests > config-shell-tests.am
 $BOOTSTRAP_SHELL ./instspc-tests.sh --generate-makefile > instspc-tests.am
 $BOOTSTRAP_SHELL ./depmod-tests.sh --generate-makefile > depmod-tests.am
 cd ..
diff --git a/tests/.gitignore b/tests/.gitignore
index c309c22..a6d0cce 100644
--- a/tests/.gitignore
+++ b/tests/.gitignore
@@ -3,6 +3,7 @@ automake-*
 defs-static
 instspc-tests.am
 parallel-tests.am
+config-shell-tests.am
 depmod-tests.am
 *.dir
 *.log
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 77c4570..f66a093 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -93,30 +93,16 @@ $(depmod_tests:.depmod=.log): depmod-tests.sh 
depmod-data.log
 $(depmod_tests):
 
 
+include $(srcdir)/config-shell-tests.am
+
+$(srcdir)/config-shell-tests.am: gen-config-shell-tests Makefile.am
+       $(AM_V_GEN)($(am__cd) $(srcdir) && $(SHELL) ./gen-config-shell-tests) 
>$@
+EXTRA_DIST += gen-config-shell-tests
+
 TEST_EXTENSIONS += .shtst
 SHTST_LOG_COMPILER = $(SHELL) $(srcdir)/config-shell-tests.sh
 EXTRA_DIST += config-shell-tests.sh
 
-# FIXME: this list might probably be made auto-generated, like
-# $(parallel_tests) is...  But would that worth the hassle?
-config_shell_tests = \
-  ar-lib-w.shtst \
-  compile-w.shtst \
-  compile2-w.shtst \
-  compile3-w.shtst \
-  compile4-w.shtst \
-  compile5-w.shtst \
-  compile6-w.shtst \
-  instsh2-w.shtst \
-  instsh3-w.shtst \
-  mkinst3-w.shtst \
-  mdate5-w.shtst \
-  mdate6-w.shtst \
-  missing-w.shtst \
-  missing2-w.shtst \
-  missing3-w.shtst \
-  missing5-w.shtst
-
 # All `*-w.shtst' tests work by sourcing the script `config-shell-tests.sh'.
 $(config_shell_tests:.shtst=.log): config-shell-tests.sh
 # This dummy declaration is required to have make actually produce
diff --git a/tests/Makefile.in b/tests/Makefile.in
index 532765e..39984b6 100644
--- a/tests/Makefile.in
+++ b/tests/Makefile.in
@@ -50,7 +50,8 @@ POST_UNINSTALL = :
 build_triplet = @build@
 host_triplet = @host@
 DIST_COMMON = README $(srcdir)/Makefile.am $(srcdir)/Makefile.in \
-       $(srcdir)/aclocal.in $(srcdir)/automake.in $(srcdir)/defs \
+       $(srcdir)/aclocal.in $(srcdir)/automake.in \
+       $(srcdir)/config-shell-tests.am $(srcdir)/defs \
        $(srcdir)/defs-static.in $(srcdir)/depmod-tests.am \
        $(srcdir)/instspc-tests.am $(srcdir)/parallel-tests.am \
        $(top_srcdir)/CheckListOfTests.am
@@ -306,8 +307,8 @@ top_builddir = @top_builddir@
 top_srcdir = @top_srcdir@
 MAINTAINERCLEANFILES = 
 EXTRA_DIST = ChangeLog-old gen-parallel-tests parallel-tests.sh \
-       instspc-tests.sh depmod-tests.sh config-shell-tests.sh \
-       $(handwritten_tests)
+       instspc-tests.sh depmod-tests.sh gen-config-shell-tests \
+       config-shell-tests.sh $(handwritten_tests)
 TEST_EXTENSIONS = .test .ptest .instspc .depmod .shtst
 # Run the tests with the shell detected at configure time.
 TEST_LOG_COMPILER = $(SHELL)
@@ -386,29 +387,13 @@ depmod_tests = depcomp-gcc3.depmod depcomp-gcc.depmod \
        depcomp-cpp.depmod depcomp-msvisualcpp.depmod \
        depcomp-msvcmsys.depmod
 DEPMOD_LOG_COMPILER = $(SHELL) $(srcdir)/depmod-tests.sh
+config_shell_tests = ar-lib-w.shtst compile-w.shtst compile2-w.shtst \
+       compile3-w.shtst compile4-w.shtst compile5-w.shtst \
+       compile6-w.shtst instsh2-w.shtst instsh3-w.shtst \
+       mdate5-w.shtst mdate6-w.shtst missing-w.shtst missing2-w.shtst \
+       missing3-w.shtst missing5-w.shtst mkinst3-w.shtst
 SHTST_LOG_COMPILER = $(SHELL) $(srcdir)/config-shell-tests.sh
 
-# FIXME: this list might probably be made auto-generated, like
-# $(parallel_tests) is...  But would that worth the hassle?
-config_shell_tests = \
-  ar-lib-w.shtst \
-  compile-w.shtst \
-  compile2-w.shtst \
-  compile3-w.shtst \
-  compile4-w.shtst \
-  compile5-w.shtst \
-  compile6-w.shtst \
-  instsh2-w.shtst \
-  instsh3-w.shtst \
-  mkinst3-w.shtst \
-  mdate5-w.shtst \
-  mdate6-w.shtst \
-  missing-w.shtst \
-  missing2-w.shtst \
-  missing3-w.shtst \
-  missing5-w.shtst
-
-
 # Some testsuite-influential variables should be overridable from the
 # test scripts, but not from the environment.
 # We want warning messages and explanations for skipped tests to go to
@@ -1395,7 +1380,7 @@ all: all-am
 
 .SUFFIXES:
 .SUFFIXES: .depmod .html .instspc .log .ptest .shtst .test
-$(srcdir)/Makefile.in:  $(srcdir)/Makefile.am $(srcdir)/parallel-tests.am 
$(srcdir)/instspc-tests.am $(srcdir)/depmod-tests.am 
$(top_srcdir)/CheckListOfTests.am $(am__configure_deps)
+$(srcdir)/Makefile.in:  $(srcdir)/Makefile.am $(srcdir)/parallel-tests.am 
$(srcdir)/instspc-tests.am $(srcdir)/depmod-tests.am 
$(srcdir)/config-shell-tests.am $(top_srcdir)/CheckListOfTests.am 
$(am__configure_deps)
        @for dep in $?; do \
          case '$(am__configure_deps)' in \
            *$$dep*) \
@@ -1785,6 +1770,25 @@ $(depmod_tests:.depmod=.log): depmod-tests.sh 
depmod-data.log
 # This dummy declaration is required to have make actually produce
 # expected log files from the `.depmod.log' suffix rule.
 $(depmod_tests):
+ar-lib-w.log: ar-lib.test
+compile-w.log: compile.test
+compile2-w.log: compile2.test
+compile3-w.log: compile3.test
+compile4-w.log: compile4.test
+compile5-w.log: compile5.test
+compile6-w.log: compile6.test
+instsh2-w.log: instsh2.test
+instsh3-w.log: instsh3.test
+mdate5-w.log: mdate5.test
+mdate6-w.log: mdate6.test
+missing-w.log: missing.test
+missing2-w.log: missing2.test
+missing3-w.log: missing3.test
+missing5-w.log: missing5.test
+mkinst3-w.log: mkinst3.test
+
+$(srcdir)/config-shell-tests.am: gen-config-shell-tests Makefile.am
+       $(AM_V_GEN)($(am__cd) $(srcdir) && $(SHELL) ./gen-config-shell-tests) 
>$@
 
 # All `*-w.shtst' tests work by sourcing the script `config-shell-tests.sh'.
 $(config_shell_tests:.shtst=.log): config-shell-tests.sh
diff --git a/tests/gen-config-shell-tests b/tests/gen-config-shell-tests
new file mode 100755
index 0000000..82477e3
--- /dev/null
+++ b/tests/gen-config-shell-tests
@@ -0,0 +1,70 @@
+#! /bin/sh
+# Generate config-shell-tests.am.
+#
+# Copyright (C) 2011 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+#
+# For each test in the $(handwritten_tests) list in this Makefile.am file,
+# that directly tests features of an automake-provided shell script from
+# the `lib/' subdirectory, define a sibling test that does likewise, but
+# using, to run said script, the configure-time $SHELL instead of the
+# default system shell /bin/sh.
+#
+# A test is considered a candidate for sibling-generation if it calls the
+# `get_shell_script' function anywhere.
+#
+# Individual tests can prevent the creation of such a sibling by explicitly
+# setting the `$test_prefer_config_shell' variable to either "yes" or "no".
+# The rationale for this is that if the variable is set to "yes", the test
+# already uses $SHELL, so that a sibling would be just a duplicate; while
+# if the variable is set to "no", the test doesn't support, or is not meant
+# to use, $SHELL to run the script under testing, and forcing it to do so
+# in the sibling would likely cause a spurious failure.
+#
+
+set -e
+
+tests=`sed -n '/^handwritten_tests =/,/^$/s/\(.*\.test\).*/\1/p' Makefile.am`
+
+if test -z "$tests"; then
+  echo "$0: failed to obtain list of tests" >&2
+  exit 1
+fi
+
+{
+  grep -l '^get_shell_script ' $tests
+  grep -l ' get_shell_script ' $tests
+} |
+LC_ALL=C sort -u |
+while read tst; do
+  grep "test_prefer_config_shell=" $tst >/dev/null || echo $tst
+done |
+sed -e 's/\.test$//' |
+{
+  echo "## Generated by gen-config-shell-tests."
+  echo "## Edit Makefile.am instead of this."
+  echo "config_shell_tests ="
+  # Since `foo-w.shtst' sources `foo.test', `foo-p.log' also depends on
+  # `foo.test'.
+  sed -e '
+    h
+    s/^/config_shell_tests += /
+    s/$/-w.shtst/
+    p
+    x
+    s/.*/&-w.log: &.test/
+  '
+}
diff --git a/tests/gen-parallel-tests b/tests/gen-parallel-tests
index 340b0e7..ed975fe 100755
--- a/tests/gen-parallel-tests
+++ b/tests/gen-parallel-tests
@@ -16,10 +16,10 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-# For each test in the TESTS list in this Makefile.am file, that itself
-# tests features of the TESTS automake interface, define a sibling
-# test that does likewise, but with the option `parallel-tests' enabled.
-# Individual tests can prevent the creation of such a sibling by
+# For each test in the $(handwritten_tests) list in this Makefile.am file,
+# that itself tests features of the TESTS automake interface, define a
+# sibling test that does likewise, but with the option `parallel-tests'
+# enabled.  Individual tests can prevent the creation of such a sibling by
 # explicitly setting the `$parallel_tests' variable to either "yes" or
 # "no".  The rationale for this is that if the variable is set to "yes",
 # the test already uses the `parallel-tests' option, so that a sibling
-- 
1.7.2.3