Re: [PATCH] {master} tests: add testcases sanity-checking the testsuite

[Ralf Wildenhues]

Hi Stefano,

* Stefano Lattarini wrote on Wed, Feb 23, 2011 at 10:20:03PM CET:
> Helper subroutines, variables and other pieces of code defined
> in the `tests/defs' and used by many testcases are non-obvious,
> and tricky to get to work portably; but until now, they weren't
> tested at all in a clear and self-contained way.
> This change should remedy to the situation.

This patch is fine with me, with some nits below.  I expect some
fallout on some systems.

Pretty nifty stuff in this patch by the way.

What does the S_ prefix stand for?  Sanity?  Well, we can keep
(or lose) our sanity over all kinds of things.  ;-)
How about 'defs-' as prefix, or maybe 'suite-'?

Thank you,
Ralf

> * tests/S_cleanup.test: New test, check removal of temporary
> test working directory by `./defs'.
> * tests/S_dir.test: New test, check that tests using `./defs'
> create a proper temporary directory, and run in it.
> * tests/S_exit.test: New test, check that, in case of failing
> commands, the correct exit status is passed to the exit trap
> installed by the `./defs' script.
> * tests/S_is_newest.test: New test, checking the `is_newest'
> subroutine.
> * tests/S_me.test: New test, checking that $me gets defined
> automatically by `tests/defs' if not set, and that it can be
> overridden from either the shell or the environment.
> * tests/S_sanity.test: New test, check that the sanity checks
> performed by the `tests/defs' script works correctly.
> * tests/S_unindent.test: New test, checking the `unindent'
> subroutine.
> * tests/Makefile.am (TESTS): Update.

> --- /dev/null
> +++ b/tests/S_cleanup.test

> +# Sanity check for the automake testsuite.
> +# Check creation/removal of temporary test working directory by `./defs'.
> +
> +. ./defs || Exit 1
> +
> +# We still need a little hack to make ./defs work outside automake's
> +# tree `tests' subdirectory.  Not a big deal.
> +sed "s|^testbuilddir=.*|testbuilddir='`pwd`'|" ../defs-static >defs-static
> +cp ../defs .
> +
> +have_simlinks=false

typo symlinks here and in some but not all places below

> +ln -s defs foo && test -h foo && have_simlinks=:

I'm not sure if this test works that way.
MSYS emulates ln -s with cp -p, might also emulate test -h.
You can find out for sure by something like this:
  echo foo > a
  ln -s a b
  echo bar > a
  grep bar b

But then again, I wonder, why you even care: on systems that emulate
symlinks, you can just go along with that.  So I guess all you'd need
to test here is whether ln -s apparently succeeds.

> +export have_simlinks # Is used also by spawned shells.
> +
> +dir=dummy.dir
> +
> +# Check that pre-test cleanup works also with directories with
> +# "null" permissions, and containing broken symlinks.
> +mkdir $dir $dir/sub
> +cd $dir
> +touch file sub/file
> +if $have_symlinks; then
> +  ln -s file symlink
> +  ln -s none brokenlink
> +fi
> +cd ..
> +chmod 000 $dir/sub/* $dir/file $dir/symlink
> +chmod 000 $dir/sub $dir
> +$SHELL -c  '. ./defs' dummy.test
> +test ! -f $dir
> +test ! -d $dir
> +test ! -r $dir
> +
> +# Check that post-test cleanup works also with directories with
> +# "null" permissions, and containing broken symlinks.
> +$SHELL -c '
> +  . ./defs || Exit 1
> +  set -e
> +  mkdir dir dir/sub
> +  cd dir
> +  touch file sub/file
> +  if $have_simlinks; then
> +    ln -s file symlink
> +    ln -s none brokenlink
> +  fi
> +  cd ..
> +  chmod 000 dir/sub/* dir/file dir/symlink
> +  chmod 000 dir/sub dir
> +' dummy.test
> +test ! -f $dir
> +test ! -d $dir
> +test ! -r $dir
> +
> +# Check that pre-test cleanup do not unduly change the permissions of

s/do/does/

> +# files to which symlinks in the temporary test directory point to.
> +if $have_simlinks; then
> +
> +  mkdir dir
> +  chmod 000 dir
> +  : > file
> +  chmod 000 file
> +
> +  mkdir $dir
> +  cd $dir
> +  ln -s ../dir ../file .
> +  cd ..
> +  $SHELL -c '. ./defs' dummy.test
> +  ls -l # For debugging.
> +  ls -l file | grep "^---------- .*file"
> +  ls -ld dir | grep "^d--------- .*dir"
> +
> +  $SHELL -c '
> +    ocwd=`pwd` || exit 1
> +    . ./defs || Exit 1
> +    ln -s "$ocwd/dir" "$ocwd/file" .
> +  ' dummy.test
> +  ls -l # For debugging.
> +  ls -l file | grep "^---------- .*file"
> +  ls -ld dir | grep "^d--------- .*dir"
> +
> +  rmdir dir
> +  rm -f file
> +
> +fi # $have_symlinks
> +
> +# Check that the cleanup trap does not remove the temporary
> +# test directory in case of test failure, skip, hard-error,
> +# or upon the receiving of a signal.

or when receiving a signal.

> +for bailout_command in \
> +  'Exit 1' \
> +  'Exit 2' \
> +  'Exit 10' \
> +  'Exit 77' \
> +  'Exit 99' \
> +  'Exit 126' \
> +  'Exit 127' \
> +  'Exit 255' \
> +  'kill -1 $$' \
> +  'kill -2 $$' \
> +  'kill -9 $$' \
> +  'kill -13 $$' \
> +  'kill -15 $$' \
> +; do
> +  $SHELL -c  ". ./defs; : > foo; $bailout_command" dummy.test && Exit 1
> +  test -f dummy.dir/foo
> +done

> --- /dev/null
> +++ b/tests/S_dir.test

> +# Sanity check for the automake testsuite.
> +# Check that tests using `./defs' create a proper temporary directory,
> +# and run in it.
> +
> +set -ex
> +
> +this=S_dir
> +outcome=:
> +pwd
> +
> +rm -f $this.tmp
> +
> +$SHELL -c "
> +  . ./defs || Exit 1
> +  : > $this.tmp
> +  case \`pwd\` in '`pwd`'/$this.dir);; *) Exit 1;; esac
> +" $this.test || outcome=false
> +
> +test -f $this.tmp && { rm -f $this.tmp; outcome=false; }
> +
> +$outcome

> --- /dev/null
> +++ b/tests/S_exit.test

> +# Sanity check for the automake testsuite.
> +# Check that. in case of failing commands, the correct exit status is
> +# passed to the exit trap installed by the `./defs' script.
> +# Also check that the `errexit' shell flag is active.
> +
> +for st in 1 2 3 4 5 77 99 126 127 128 129 130 255; do
> +
> +  echo "* Try: Exit $st"

ramping up output markup again?  ;->

> +  $SHELL -c  ". ./defs; Exit $st; :"
> +  rc=$?
> +  echo "* rc=$rc"
> +  echo
> +  test $rc -eq $st || exit 1
> +
> +  echo "* Try: sh -c 'exit $st'"
> +  $SHELL -c  ". ./defs; sh -c 'exit $st'; :"
> +  rc=$?
> +  echo "* rc=$rc"
> +  echo
> +  test $rc -eq $st || exit 1
> +
> +done
> +
> +echo "* Try: non-existent-program"
> +$SHELL -c  ". ./defs; non-existent-program; :"
> +rc=$?
> +echo "* rc=$rc"
> +echo
> +test $rc -eq 127 || exit 1
> +
> +for sig in 1 2 13 15; do
> +
> +  echo "* Try: kill -$sig \$\$"
> +  $SHELL -c  ". ./defs; kill -$sig \$\$; :"
> +  rc=$?
> +  echo "* rc=$rc"
> +  echo
> +  test $rc -eq 99 || exit 1
> +
> +done

> --- /dev/null
> +++ b/tests/S_is_newest.test

> +# Sanity check for the automake testsuite.
> +# Check the `is_newest' subroutine.
> +
> +. ./defs || Exit 1
> +
> +: > a
> +$sleep
> +: > b
> +: > c
> +
> +stat a b c || : # for debugging

stat is not portable.  You knew that already, I guess.

> +is_newest c a
> +is_newest b a
> +is_newest a b && Exit 1
> +is_newest c b
> +is_newest c c
> +is_newest c a b c
> +
> +touch -r c d || Exit 77

Why the || Exit 77 here?

> +stat c d || : # for debugging
> +
> +is_newest c d
> +is_newest d c

You cannot be sure about the latter one.  See 'info Autoconf --index
touch'.

> --- /dev/null
> +++ b/tests/S_me.test

> +# Sanity check for the automake testsuite.
> +# Make sure that $me gets automatically defined by `./defs'.
> +
> +set -ex
> +
> +: ${SHELL=/bin/sh}
> +
> +$SHELL -c '. ./defs && echo me=$me' foo-bar-.test | grep '^me=foo-bar-$'
> +$SHELL -c '. ./defs && echo me=$me' _foo__bar.test | grep '^me=_foo__bar$'
> +$SHELL -c '. ./defs && echo me=$me' 012.test | grep '^me=012$'
> +$SHELL -c '. ./defs && echo me=$me' foo.bar | grep '^me=foo\.bar$'

> --- /dev/null
> +++ b/tests/S_sanity.test

> +# Sanity check for the automake testsuite.
> +# Test the sanity checks performed by the `defs' script.  Also check that
> +# it can be used by duplicating some of infrastructure of the automake's
> +# tree `tests' subdirectory.

I have a hard time understanding this sentence.  How about

  Also check that we can use `defs' elsewhere, when we duplicate some
  of the infrastructure from the automake/tests subdirectory.

> +. ./defs || Exit 1
> +
> +# Avoid to confuse traces from children processed with our own traces.

Do not confuse traces from child processes ...

> +show_stderr()

space before paren

> +{
> +  sed 's/^/ | /' stderr >&2
> +}
> +
> +if $SHELL -c '. ../defs' dummy.test 2>stderr; then
> +  show_stderr
> +  Exit 1
> +else
> +  show_stderr
> +  grep 'defs-static: not found in current directory' stderr
> +fi
> +
> +sed 's|^testsrcdir=.*|testsrcdir=foo|' ../defs-static > defs-static
> +if $SHELL -c '. ../defs' dummy.test 2>stderr; then
> +  show_stderr
> +  Exit 1
> +else
> +  show_stderr
> +  grep 'foo/defs-static\.in not found.*check \$testsrcdir' stderr
> +fi
> +
> +sed 's|^testbuilddir=.*|testbuilddir=foo|' ../defs-static > defs-static
> +if $SHELL -c '. ../defs' dummy.test 2>stderr; then
> +  show_stderr
> +  Exit 1
> +else
> +  show_stderr
> +  grep 'foo/defs-static not found.*check \$testbuilddir' stderr
> +fi
> +
> +# We still need a little hack to make ./defs work outside automake's
> +# tree `tests' subdirectory.  Not a big deal.
> +sed "s|^testbuilddir=.*|testbuilddir='`pwd`'|" ../defs-static >defs-static
> +# Redefining *srcdir and *builddir variables in the environment shouldn't
> +# cause problems
> +env \
> +  builddir=bad-dir srcdir=bad-dir \
> +  top_builddir=bad-dir top_srcdir=bad-dir \
> +  abs_builddir=bad-dir abs_srcdir=bad-dir \
> +  abs_top_builddir=bad-dir abs_top_srcdir=bad-dir \
> +  $SHELL -c  '. ../defs && echo "!OK!" > ../foo' dummy.test
> +$FGREP '!OK!' foo
> +
> +:

> --- /dev/null
> +++ b/tests/S_unindent.test

> +# Sanity check for the automake testsuite.
> +# Check the `unindent' subroutine.
> +
> +. ./defs || Exit 1
> +
> +#------------------------------------------------------------------
> +
> +: Leading spaces
> +
> +cat > input <<END
> +  1
> +   2
> +3
> + 4
> +${tab}5
> +${tab}  6
> +  6${sp}
> +7${sp}
> +  8${sp}${sp}
> +9${sp}${sp}
> +  10${tab}
> +11${tab}
> +  12${sp}${tab}
> +13${sp}${tab}
> +  14 this${tab}with${tab}multiple fields${sp}
> +15 and   ${tab}${tab}this too${tab}
> + 16 and also this
> +${sp}${sp}
> +${sp}
> +${tab}
> +
> +last line
> +END
> +
> +cat > exp <<END
> +1
> + 2
> +3
> + 4
> +${tab}5
> +${tab}  6
> +6${sp}
> +7${sp}
> +8${sp}${sp}
> +9${sp}${sp}
> +10${tab}
> +11${tab}
> +12${sp}${tab}
> +13${sp}${tab}
> +14 this${tab}with${tab}multiple fields${sp}
> +15 and   ${tab}${tab}this too${tab}
> + 16 and also this
> +
> +${sp}
> +${tab}
> +
> +last line
> +END
> +
> +unindent input > got
> +
> +diff exp got
> +
> +#------------------------------------------------------------------
> +
> +: Leading tab
> +
> +cat > input <<END
> +${tab}1
> +${tab} 2
> +3
> + 4
> +  5
> +    6
> +        7
> + ${tab}8
> +${tab}${tab}9
> +${tab}10${tab}
> +${tab}11${sp}
> +12${tab}
> +13${sp}
> +${tab}14 this  with${tab}multiple fields${sp}
> +15 and   ${tab}${tab}this too${tab}
> + 16 and also this
> +${tab}
> +${sp}
> +${sp}${tab}
> +
> +last line
> +END
> +
> +cat > exp <<END
> +1
> + 2
> +3
> + 4
> +  5
> +    6
> +        7
> + ${tab}8
> +${tab}9
> +10${tab}
> +11${sp}
> +12${tab}
> +13${sp}
> +14 this  with${tab}multiple fields${sp}
> +15 and   ${tab}${tab}this too${tab}
> + 16 and also this
> +
> +${sp}
> +${sp}${tab}
> +
> +last line
> +END
> +
> +unindent input > got
> +
> +diff exp got
> +
> +#------------------------------------------------------------------
> +
> +: No leading whitespace
> +
> +cat > input <<END
> +1
> + 2
> +  3
> +        4
> +${tab}5
> +${tab} 6
> + ${tab}7
> +${tab}${tab}8
> +9${sp}
> + 10${tab}
> +${tab}10${sp}${sp}
> +14 this with${tab}multiple  fields${sp}
> + 15 and this too${tab}
> +${tab}16 and also this
> +${tab}
> +${sp}
> +
> +last line
> +END
> +
> +cp input exp
> +
> +unindent input > got
> +
> +diff exp got
> +
> +#------------------------------------------------------------------
> +
> +: Leading empty lines are ignored [1]
> +
> +cat > input <<END
> +
> +
> + foo
> + bar
> +   quux
> +END
> +
> +cat > exp <<END
> +
> +
> +foo
> +bar
> +  quux
> +END
> +
> +unindent input > got
> +
> +diff exp got
> +
> +#------------------------------------------------------------------
> +
> +: Leading empty lines are ignored [2]
> +
> +cat > input <<END
> +
> +
> +foo
> +bar
> +   quux
> +END
> +
> +cat > exp <<END
> +
> +
> +foo
> +bar
> +   quux
> +END
> +
> +unindent input > got
> +
> +diff exp got
> +
> +#------------------------------------------------------------------
> +
> +: