[Automake-commit] [SCM] GNU Automake branch, tap-testsuite-work, updated. v1.11-1110-g2735939

[Stefano Lattarini]

This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "GNU Automake".

http://git.sv.gnu.org/gitweb/?p=automake.git;a=commitdiff;h=273593955a62bbbd7e20b91367c66b6e547d4882

The branch, tap-testsuite-work has been updated
       via  273593955a62bbbd7e20b91367c66b6e547d4882 (commit)
       via  c7fa87264291f611fdc63bd4a49913f98ec5b944 (commit)
       via  9f8d51bd31b870b4af9dac5b4030bf3a5c155262 (commit)
       via  596ae38451e269e40e3b30ab6f6f016bd9076c04 (commit)
       via  ff37c9ba221b5b6e2bb365a438617113c89be6c9 (commit)
       via  265671e4e2cbd32efaf71595db19b37f97290b69 (commit)
      from  0bc42aa61e89dcbde7dc64f932ec795f383f9211 (commit)

Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.

- Log -----------------------------------------------------------------
commit 273593955a62bbbd7e20b91367c66b6e547d4882
Merge: 0bc42aa c7fa872
Author: Stefano Lattarini <address@hidden>
Date:   Sat Aug 6 22:15:50 2011 +0200

    Merge branch 'test-protocols' into tap-testsuite-work
    
    * test-protocols:
      tap driver: handle signals received by the tests being run
      test driver: a preparatory refactoring (2)
      tap driver: a preparatory refactoring (1)
      cosmetics: typofix in the name of a function in the TAP driver
      tap: scripts with a SKIP plan but with exit status != 0 must error

-----------------------------------------------------------------------

Summary of changes:
 ChangeLog                                          |   52 ++++++++++++
 doc/automake.texi                                  |    3 +-
 lib/tap-driver                                     |   87 +++++++++++++++-----
 tests/Makefile.am                                  |    1 +
 tests/Makefile.in                                  |    1 +
 tests/tap-signal.test                              |   36 ++++++--
 ...hard-error.test => tap-skip-whole-badexit.test} |   34 ++++----
 7 files changed, 166 insertions(+), 48 deletions(-)
 copy tests/{tap-no-disable-hard-error.test => tap-skip-whole-badexit.test} 
(59%)

diff --git a/ChangeLog b/ChangeLog
index be562fb..1366010 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,5 +1,57 @@
 2011-08-06  Stefano Lattarini  <address@hidden>
 
+       tap driver: handle signals received by the tests being run
+       * lib/tap-driver (get_test_exit_message): Also deal with signals,
+       by using the `wait' method of the TAP::Parser object instead of
+       the `exit' method.  This required the use of the standard perl
+       module `POSIX'.
+       * doc/automake.texi (Use TAP with the Automake test harness):
+       Document that `--ignore-exit' has effect also on terminating
+       signals.  Add a "synchronizing" comment that references the tests
+       'tap-exit.test' and 'tap-signal.test'.
+       * tests/tap-signal.test: Extend and adjust.
+
+2011-08-06  Stefano Lattarini  <address@hidden>
+
+       test driver: a preparatory refactoring (2)
+       * lib/tap-driver (finish): Move code to fetch the message about
+       the exit status of the test being run to ...
+       (get_test_exit_message): ... this new subroutine.
+
+2011-08-06  Stefano Lattarini  <address@hidden>
+
+       tap driver: a preparatory refactoring (1)
+       * lib/tap-driver (finish): Do not flush the TAP stream nor
+       fetch the exit status of the TAP producer if running with
+       the `--ignore-exit' option.  Obviously, don't try to write
+       such exit status in the test log anymore.
+
+2011-08-06  Stefano Lattarini  <address@hidden>
+
+       cosmetics: typofix in the name of a function in the TAP driver
+       * lib/tap-driver (testuite_error): Renamed ...
+       (testsuite_error): ... to this, and minor related reformatting.
+
+2011-08-06  Stefano Lattarini  <address@hidden>
+
+       tap: scripts with a SKIP plan but with exit status != 0 must error
+       This change has been motivated by Automake's own testsuite.  Some
+       TAP tests there were erroring out (due to unexpected, unhandled
+       failures) before having encountered TAP result, so that the
+       simple-minded shell library implementing TAP generation ended up
+       printing a "1..0" trailing test plan; this caused the script to be
+       reported as a SKIP rather than an ERROR -- a nasty false negative.
+       * lib/tap-driver: Add prototypes for each subroutine, to free up
+       the order in which they can be defined and called.
+       (main): Move the code checking for a bad exit status of the TAP
+       producer ...
+       (finish): ... here, and flush the TAP stream to ensure that the
+       parser always obtains the producer's exit status.
+       * tests/tap-skip-whole-badexit.test: New test.
+       * tests/Makefile.am (tap_with_common_setup_tests): Add it.
+
+2011-08-06  Stefano Lattarini  <address@hidden>
+
        tap: fix whitespace munging of diagnostic messages
        * lib/tap-driver (extract_tap_comment): Pass the `g' flag to the
        substitution operator, to strip also trailing whitespaces.  Fixes
diff --git a/doc/automake.texi b/doc/automake.texi
index c63742d..6be5c2d 100644
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -9646,10 +9646,11 @@ supports the following options, whose names are chosen 
for enhanced
 compatibility with the @command{prove} utility.
 
 @table @option
address@hidden Keep in sync with 'tap-exit.test' and 'tap-signal.test'.
 @item --ignore-exit
 Causes the test driver to ignore the exit status of the test scripts;
 by default, the driver will report an error if the script exit with a
-non-zero status.
+non-zero status.  This option has effect also
 @item --comments
 Instruct the test driver to display TAP diagnostic (i.e., lines beginning
 with the @samp{#} character) in the testsuite progress output too; by
diff --git a/lib/tap-driver b/lib/tap-driver
index d59ae01..1f5a271 100755
--- a/lib/tap-driver
+++ b/lib/tap-driver
@@ -83,6 +83,35 @@ Getopt::Long::GetOptions (
     'ignore-exit' => sub { $cfg{"ignore-exit"} = 1; },
   ) or exit 1;
 
+# ------------- #
+#  Prototypes.  #
+# ------------- #
+
+sub add_test_result ($);
+sub bool_opt ($$);
+sub colored ($$);
+sub copy_in_global_log ();
+sub decorate_result ($);
+sub extract_tap_comment ($);
+sub finish ();
+sub get_global_test_result ();
+sub get_tap_line ();
+sub get_test_exit_message ();
+sub get_test_results ();
+sub handle_tap_bailout ($);
+sub handle_tap_plan ($);
+sub handle_tap_test ($);
+sub main (@);
+sub must_recheck ();
+sub peek_tap_line ();
+sub report ($;$);
+sub start (@);
+sub stringify_test_result ($);
+sub testsuite_error ($);
+sub unget_tap_line ($);
+sub write_test_results ();
+sub yn ($);
+
 # -------------- #
 #  Subroutines.  #
 # -------------- #
@@ -202,8 +231,37 @@ sub start (@)
   $parser->ignore_exit(1) if $cfg{"ignore-exit"};
 }
 
+sub get_test_exit_message ()
+{
+  # Flush all the remaining TAP stream, so that we can obtain the
+  # exit status of the TAP producer.
+  do {} while defined get_tap_line ();
+  my $wstatus = $parser->wait;
+  # Return an undefined value if the producer exited with success.
+  return unless $wstatus;
+  # Otherwise, determine whether it exited with error or was terminated
+  # by a signal.
+  use POSIX qw (WIFEXITED WEXITSTATUS WIFSIGNALED WTERMSIG);
+  if (WIFEXITED ($wstatus))
+       {
+      return sprintf "exited with status %d", WEXITSTATUS ($wstatus);
+       }
+  elsif (WIFSIGNALED ($wstatus))
+       {
+      return sprintf "terminated by signal %d", WTERMSIG ($wstatus);
+       }
+  else
+       {
+         return "terminated abnormally";
+       }
+}
+
 sub finish ()
 {
+  if (!$cfg{"ignore-exit"} and my $msg = get_test_exit_message)
+    {
+      testsuite_error $msg;
+    }
   write_test_results;
   close LOG or die "closing $log_file: $!\n";
   exit 0;
@@ -287,7 +345,7 @@ sub report ($;$)
   print $result . $msg;
 }
 
-sub testuite_error ($)
+sub testsuite_error ($)
 {
   report "ERROR", "- $_[0]";
 }
@@ -326,12 +384,12 @@ sub handle_tap_test ($)
 sub handle_tap_plan ($)
 {
   my $plan = shift;
-  testuite_error "multiple test plans" if $plan_seen;
+  testsuite_error "multiple test plans" if $plan_seen;
   $plan_seen = 1;
   # TAP plan must be either in the first or in the last line.
   if ($lineno > 1 && peek_tap_line)
     {
-      testuite_error "test plan in middle of output";
+      testsuite_error "test plan in middle of output";
       return;
     }
   # Nothing more to do, unless the plan contains a SKIP directive.
@@ -347,7 +405,7 @@ sub handle_tap_bailout ($)
 {
   my ($bailout, $msg) = ($_[0], "Bail out!");
   $msg .= " " . $bailout->explanation if $bailout->explanation;
-  testuite_error $msg;
+  testsuite_error $msg;
   finish;
 }
 
@@ -391,30 +449,15 @@ sub main (@)
     }
   if (!$plan_seen)
     {
-      testuite_error "missing test plan";
+      testsuite_error "missing test plan";
     }
   elsif ($parser->tests_planned != $parser->tests_run)
     {
       my ($planned, $run) = ($parser->tests_planned, $parser->tests_run);
       my $bad_amount = $run > $planned ? "many" : "few";
-      testuite_error (sprintf "too %s tests run (expected %d, got %d)",
-                              $bad_amount, $planned, $run);
+      testsuite_error (sprintf "too %s tests run (expected %d, got %d)",
+                               $bad_amount, $planned, $run);
     }
-  # TODO: we should probably use $parser->wait here, to catch signals too
-  if ($parser->exit != 0)
-  {
-    my $msg = sprintf "exited with status %d", $parser->exit;
-    if ($cfg{"ignore-exit"})
-      {
-        # Log the exit status of the script anyway, even if it is not
-        # considered to be an error, to help debugging.
-        print "INFO: $test_script_name - $msg\n";
-      }
-    else
-      {
-        testuite_error $msg;;
-      }
-  }
   finish;
 }
 
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 4fd4b79..1865bbd 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -1184,6 +1184,7 @@ tap-recheck-logs.test \
 tap-skip-whole-whitespace.test \
 tap-skip-whole.test \
 tap-skip-whole-lastline.test \
+tap-skip-whole-badexit.test \
 tap-todo-skip-together.test \
 tap-todo-skip-whitespace.test \
 tap-todo-skip.test \
diff --git a/tests/Makefile.in b/tests/Makefile.in
index a0c553f..e28d6ed 100644
--- a/tests/Makefile.in
+++ b/tests/Makefile.in
@@ -1409,6 +1409,7 @@ tap-recheck-logs.test \
 tap-skip-whole-whitespace.test \
 tap-skip-whole.test \
 tap-skip-whole-lastline.test \
+tap-skip-whole-badexit.test \
 tap-todo-skip-together.test \
 tap-todo-skip-whitespace.test \
 tap-todo-skip.test \
diff --git a/tests/tap-signal.test b/tests/tap-signal.test
index fb75c83..ce75ddd 100755
--- a/tests/tap-signal.test
+++ b/tests/tap-signal.test
@@ -36,14 +36,32 @@ chmod a+x *.test
 
 . "$testsrcdir"/tap-setup.sh || fatal_ "sourcing tap-setup.sh"
 
-for append in '' 'TEST_LOG_DRIVER_FLAGS = --ignore-exit'; do
-  echo "$append" >> Makefile
-  $MAKE check >stdout && { cat stdout; Exit 1; }
-  cat stdout
-  count_test_results total=8 pass=4 fail=0 xpass=0 xfail=0 skip=0 error=4
-  for sig in 1 2 13 15; do
-    grep "^ERROR: signal-$sig\\.test - terminated by signal.*" stdout
-  done
-done
+signal_caught ()
+{
+  numeric=$1
+  symbolic=$2
+  sig_re="((SIG)?$symbolic|$numeric)"
+  tst_re="signal-$numeric\\.test"
+  $EGREP "^ERROR: $tst_re - terminated by signal $sig_re$" stdout
+}
+
+all_signals_caught ()
+{
+  # This are the only signals that are portably trappable.
+  signal_caught  1 HUP
+  signal_caught  2 INT
+  signal_caught 13 PIPE
+  signal_caught 15 TERM
+}
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=8 pass=4 fail=0 xpass=0 xfail=0 skip=0 error=4
+all_signals_caught
+
+echo 'TEST_LOG_DRIVER_FLAGS = --ignore-exit' >> Makefile
+$MAKE check >stdout || { cat stdout; Exit 1; }
+cat stdout
+count_test_results total=4 pass=4 fail=0 xpass=0 xfail=0 skip=0 error=0
 
 :
diff --git a/tests/tap-no-disable-hard-error.test 
b/tests/tap-skip-whole-badexit.test
similarity index 59%
copy from tests/tap-no-disable-hard-error.test
copy to tests/tap-skip-whole-badexit.test
index 678e184..76ba692 100755
--- a/tests/tap-no-disable-hard-error.test
+++ b/tests/tap-skip-whole-badexit.test
@@ -15,36 +15,38 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 # TAP support:
-#  - "Bail out!" magic and TAP parse errors are not disabled nor turned
-#    into simple failures by the definition DISABLE_HARD_ERRORS.
+#  - an exit status != 0 of a test script causes an hard error, even if
+#    the last line of output is a "SKIP plan" (e.g., "1..0 # SKIP").
 
 parallel_tests=yes
 . ./defs || Exit 1
 
-cat > Makefile.am << 'END'
-DISABLE_HARD_ERRORS = yes
-TEST_LOG_COMPILER = cat
-TESTS = bail.test few.test noplan.test
-END
+echo TESTS = one.test two.test > Makefile.am
 
 . "$testsrcdir"/tap-setup.sh || fatal_ "sourcing tap-setup.sh"
 
-cat > bail.test <<END
-1..1
-Bail out!
+cat > one.test <<'END'
+#!/bin/sh
+echo '1..0 # SKIP'
+exit 1
 END
 
-cat > few.test <<END
-1..1
+cat > two.test <<'END'
+#!/bin/sh
+echo '1..0'
+exit 22
 END
 
-cat > noplan.test <<END
-# nothing here
-END
+chmod a+x one.test two.test
 
 $MAKE check >stdout && { cat stdout; Exit 1; }
 cat stdout
 
-count_test_results total=3 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=3
+# The 'prove' utility reports both the skip and the non-zero exit status,
+# so we do the same.
+count_test_results total=4 pass=0 fail=0 xpass=0 xfail=0 skip=2 error=2
+
+grep '^ERROR: one\.test - exited with status 1$' stdout
+grep '^ERROR: two\.test - exited with status 22$' stdout
 
 :


hooks/post-receive
-- 
GNU Automake