unattended testing on w32

[Ralf Wildenhues]

Hello,

both the old and the new Libtool testsuites contain some tests that
cause programs to fail to start, intentionally.  As we add more test
coverage, the number of such tests invariably increases, too.

Now, a failed program execution on MSYS will cause a popup window
notifying about the situation, requiring you to acknowledge it, and
maybe even to avoid sending an online report to some software vendor
we are not affiliated with.

Efficient testing is hard if it requires human attention, so let's
make it easy to run only the unattended parts, or only the complement.
The patch below introduces a new Autotest keyword 'interactive' for
those tests that may require user intervention (on w32).  It also adds
check-(non)interactive rules that run only the respective set of tests
in the new testsuite.

Some documentation about all used Autotest keywords is added, too, but
I'm not so sure whether that is a wise thing to do as it requires us to
be strict in test group naming (-k aka. --keyword also matches words in
the test group names).

I'm still pondering how to best treat the old testsuite here.  IIRC then
at least the demo-relink and depdemo-relink tests failed (any others?),
but they require a handful of other demo-*/depdemo-* tests to run first.

What do you think of this approach?

Note that, since Autotest also uses the test group names for matching
keywords, that means you cannot use the documented keywords freely in
new test group names except where appropriate.  I wonder if that should
be documented as well.

Thanks,
Ralf

    Testsuite keyword 'interactive' and check-* rules.
    
    * Makefile.am (testsuite_deps, testsuite_deps_uninstalled):
    New variables.
    (check-local, installcheck-local): Use them.
    (check-interactive, check-noninteractive): New rules.
    * tests/link-order2.at (Link order of deplibs),
    tests/static.at (static linking flags for programs): Add keyword
    `interactive'.
    * doc/libtool.texi (Test descriptions): Document all keywords
    used in the Libtool test suite.
    * NEWS: Update.

diff --git a/Makefile.am b/Makefile.am
index 71fd91b..7f2f5c2 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -546,19 +546,38 @@ DISTCLEANFILES += tests/atconfig
 
 CD_TESTDIR     = abs_srcdir=`$(lt__cd) $(srcdir) && pwd`; cd tests
 
+testsuite_deps = tests/atconfig $(srcdir)/$(TESTSUITE)
+testsuite_deps_uninstalled = $(testsuite_deps) libltdl/libltdlc.la 
$(bin_SCRIPTS)
+
 # Hook the test suite into the check rule
-check-local: tests/atconfig $(srcdir)/$(TESTSUITE) libltdl/libltdlc.la 
$(bin_SCRIPTS)
+check-local: $(testsuite_deps_uninstalled)
        $(CD_TESTDIR); \
        CONFIG_SHELL="$(SHELL)" $(SHELL) $$abs_srcdir/$(TESTSUITE) \
          $(TESTS_ENVIRONMENT) $(BUILDCHECK_ENVIRONMENT) $(TESTSUITEFLAGS)
 
 # Run the test suite on the *installed* tree.
-installcheck-local: tests/atconfig $(srcdir)/$(TESTSUITE)
+installcheck-local: $(testsuite_deps)
        $(CD_TESTDIR); \
        CONFIG_SHELL="$(SHELL)" $(SHELL) $$abs_srcdir/$(TESTSUITE) \
          $(TESTS_ENVIRONMENT) $(INSTALLCHECK_ENVIRONMENT) $(TESTSUITEFLAGS) \
          AUTOTEST_PATH="$(exec_prefix)/bin"
 
+# Run only noninteractive parts of the new testsuite.
+check-noninteractive: $(testsuite_deps_uninstalled)
+       $(CD_TESTDIR); \
+       CONFIG_SHELL="$(SHELL)" $(SHELL) $$abs_srcdir/$(TESTSUITE) \
+         $(TESTS_ENVIRONMENT) $(BUILDCHECK_ENVIRONMENT) \
+         -k !interactive INNER_TESTSUITEFLAGS=",!interactive" \
+         $(TESTSUITEFLAGS)
+
+# Run only interactive parts of the new testsuite.
+check-interactive: $(testsuite_deps_uninstalled)
+       $(CD_TESTDIR); \
+       CONFIG_SHELL="$(SHELL)" $(SHELL) $$abs_srcdir/$(TESTSUITE) \
+         $(TESTS_ENVIRONMENT) $(BUILDCHECK_ENVIRONMENT) \
+         -k interactive -k recursive INNER_TESTSUITEFLAGS=",interactive" \
+         $(TESTSUITEFLAGS)
+
 # We need to remove any file droppings left behind by testsuite
 clean-local: clean-local-legacy
        -$(CD_TESTDIR); \
diff --git a/NEWS b/NEWS
index 899e085..ac15b55 100644
--- a/NEWS
+++ b/NEWS
@@ -15,6 +15,9 @@ New in 2.2.8 2009-??-??: git version 2.2.7a, Libtool team:
   - New libtool command line flag --no-finish, to use as an optimization to
     avoid multiple finish mode commands stemming from multiple installs into
     the same library directory.
+  - New convenience make targets `check-noninteractive' to avoid long testsuite
+    runs on Windows with popup windows in the middle, and `check-interactive'
+    for the complement set of tests.
 
 * Changes in supported systems or compilers:
 
diff --git a/doc/libtool.texi b/doc/libtool.texi
index 176f797..b4b9f37 100644
--- a/doc/libtool.texi
+++ b/doc/libtool.texi
@@ -5175,6 +5175,74 @@ directory, except that Fortran 90 is used in combination 
with the
 
 @end table
 
+The new, Autotest-based test suite uses keywords to classify certain
+test groups:
+
address@hidden @samp
address@hidden CXX
address@hidden F77
address@hidden FC
address@hidden GCJ
+The test group exercises one of these @command{libtool} language tags.
+
address@hidden autoconf
address@hidden automake
+These keywords denote that the respective external program is needed
+by the test group.  The tests are typically skipped if the program is
+not installed.  The @samp{automake} keyword may also denote use of the
address@hidden program.
+
address@hidden interactive
+This test group may require user interaction on some systems.  Typically,
+this means closing a popup window about a DLL load error on Windows.
+
address@hidden libltdl
+Denote that the @file{libltdl} library is exercised by the test group.
+
address@hidden libtool
address@hidden libtoolize
+Denote that the @command{libtool} or @command{libtoolize} scripts are
+exercised by the test group, respectively.
+
address@hidden recursive
+Denote that this test group may recursively re-invoke the test suite
+itself, with changed settings and maybe a changed @command{libtool}
+script.  You may use the @env{INNER_TESTSUITEFLAGS} variable to pass
+additional settings to this recursive invocation.  Typically, recursive
+invocations delimit the set of tests with another keyword, for example
+by passing @code{-k libtool} right before the expansion of the
address@hidden variable (without an intervening space, so
+you get the chance for further delimitation).
+
+Test groups with the keyword @samp{recursive} should not be denoted with
+keywords, in order to avoid infinite recursion.  As a consequence,
+recursive test groups themselves should never require user interaction,
+while the test groups they invoke may do so.
address@hidden table
+
+For example, in order to avoid any interactive test groups, you could
+use this:
+
address@hidden
+make check-local \
+  TESTSUITEFLAGS='-k !interactive INNER_TESTSUITEFLAGS=",!interactive"'
address@hidden smallexample
+
address@hidden
+while to run only those test groups, you would use this:
+
address@hidden
+make check-local \
+  TESTSUITEFLAGS='-k interactive -k recursive 
INNER_TESTSUITEFLAGS=,interactive'
address@hidden smallexample
+
address@hidden
+but the convenience targets @samp{check-interactive} and
address@hidden avoid needing to remember these complex
+commands, in addition to also disabling interactive tests in the old
+test suite.
+
+
 @node When tests fail
 @subsection When tests fail
 @cindex failed tests
diff --git a/tests/link-order2.at b/tests/link-order2.at
index 5f1c936..4bf9716 100644
--- a/tests/link-order2.at
+++ b/tests/link-order2.at
@@ -45,6 +45,7 @@
 
 AT_SETUP([Link order of deplibs])
 AT_KEYWORDS([libtool])
+AT_KEYWORDS([interactive])dnl running 'wrong' may cause a popup window.
 
 eval "`$LIBTOOL --config | $EGREP '^(shlibpath_var|allow_undefined_flag)='`"
 
diff --git a/tests/static.at b/tests/static.at
index 1a09c56..b5fde17 100644
--- a/tests/static.at
+++ b/tests/static.at
@@ -67,6 +67,7 @@
 
 AT_SETUP([static linking flags for programs])
 AT_KEYWORDS([libtool])
+AT_KEYWORDS([interactive])dnl Some of the exec_fail test cause popups with 
MinGW.
 
 LDFLAGS="$LDFLAGS -no-undefined"
 prefix=`pwd`/inst