Re: [PATCH] {master} tests: add testcases sanity-checking the testsuite

[Stefano Lattarini]

On Friday 25 February 2011, Ralf Wildenhues wrote:
> * Stefano Lattarini wrote on Thu, Feb 24, 2011 at 11:25:18PM CET:
> > On Thursday 24 February 2011, Ralf Wildenhues wrote:
> > > What does the S_ prefix stand for?  Sanity?
> > >
> > Yes (or also "self-check").  It also ensures (when LC_COLLATE=C or
> > equivalent) that the new testcases are listed earlier by e.g. ls or
> > shell wildcarding, and that are run early, (mostly) before all the
> > other testcases.
> > 
> > > Well, we can keep (or lose) our sanity over all kinds of things.  ;-)
> > > How about 'defs-' as prefix, or maybe 'suite-'?
> > >
> > Weren't you the one fond of shorter test names? ;-)
> 
> Yes, but only those I already knew.  ;->
> 
> > Kidding aside, I'd rather keep the new testcases standing out more
> > clearly (as they do with the current `S_' prefix).  So for the moment
> > I haven't renamed them.  But feel free to advance other proposals, or
> > to override my decision if the current naming really bothers you.
> 
> How about 'self-' as prefix, or maybe even 'self-check-' then?
>
'self-check-' is definitely good enough.  I went for it.

> With S_, there is really no clue to understand, and also underscores
> are harder to type.  You might laugh at this argument, but look around
> GNU code how it tends to prefer hyphen over underscore whenever possible
> (and rejects capitalized names).
> 
> > I'll push in 72 hours if there are no further inputs or objections.
> 
> Well, one issue I noticed, and one general comment:
> Please try your tests when $sh_errexit_works is "no" (you can fake
> that), fix the fallout.
>
Well caught.  Done in the attached squash-in (which BTW also fixes
an unrelated bug in `exit.test' due to a weird Korn shell interation
with SIGINT).

> Generally, you can't test the testsuite internals with absolute
> certainty from within.  Say, there is a bug that lets every test
> pass: you will not notice it without some manual looking at logs.
>
Well, the test `S_exit.test' (now `self-check-exit.test') should help
to avoid such a situation (not with a 100% certainity, but such a
certainity has never been my aim anyway).   Also, it's quite usual
for me to run tests from the command line, so I guess I might notice
that something is wrong with the trestsuite by looking at the output
on the screen.

> (XFAILing tests can be of help here, but only if the suite is not
> buggy to mis-detect them too.)
> 
> Thanks,
> Ralf
> 

I'll push in 24 hours if there's no further objection.

Thanks,
    Stefano

diff --git a/tests/self-check-cleanup.test b/tests/self-check-cleanup.test
index 7fdde51..92a203e 100755
--- a/tests/self-check-cleanup.test
+++ b/tests/self-check-cleanup.test
@@ -19,6 +19,11 @@
 
 . ./defs || Exit 1
 
+if test x"$sh_errexit_works" != x"yes"; then
+  echo "$me: the shell can't have a working exit trap with 'set -e'" >&2
+  Exit 77
+fi
+
 # We still need a little hack to make ./defs work outside automake's
 # tree `tests' subdirectory.  Not a big deal.
 sed "s|^testbuilddir=.*|testbuilddir='`pwd`'|" ../defs-static >defs-static
diff --git a/tests/self-check-exit.test b/tests/self-check-exit.test
index 25a7219..73f3ea5 100755
--- a/tests/self-check-exit.test
+++ b/tests/self-check-exit.test
@@ -19,6 +19,8 @@
 # passed to the exit trap installed by the `./defs' script.
 # Also check that the `errexit' shell flag is active.
 
+. ./defs-static || exit 99
+
 for st in 1 2 3 4 5 77 99 126 127 128 129 130 255; do
 
   echo "* Try: Exit $st"
@@ -47,11 +49,31 @@ test $rc -eq 127 || exit 1
 for sig in 1 2 13 15; do
 
   echo "* Try: kill -$sig \$\$"
+  if test $sig -eq 2; then
+    # Some Korn shells might otherwise get a spurious SIGINT
+    # signal when one is sent to the child $SHELL.
+    trap : 2
+  fi
   $SHELL -c  ". ./defs; kill -$sig \$\$; :"
   rc=$?
+  if test $sig -eq 2; then
+    # Reset default SIGINT handler as portably as possible.
+    trap 2 || trap - 2
+  fi
   echo "* rc=$rc"
   echo
-  test $rc -eq 99 || exit 1
+  if test x"$sh_errexit_works" = x"yes"; then
+    # The exit trap should turn into an hard errors any failure
+    # caused by signals.
+    test $rc -eq 99 || exit 1
+  else
+    # The exit trap is not installed, so that the shell should exit
+    # with status 128+n when receiving signal number n.  But don't
+    # be too strict in the check, as POSIX only says that "The exit
+    # status of a command that terminated because it received a
+    # signal shall be reported as greater than 128".
+    test $rc -gt 128 || exit 1
+  fi
 
 done