Re: [PATCH] {maint} tests: new subroutines for test skipping/failing

[Ralf Wildenhues]

Hello Stefano,

you've already applied this patch, but I have a couple of nits anyway:

* Stefano Lattarini wrote on Mon, Jan 24, 2011 at 04:04:30PM CET:
> Subject: [PATCH] coverage: more tests on simple and parallel test drivers
> 
> * tests/parallel-tests-subdir.test: New test.
> * tests/check-exported-srcdir.test: Likewise.
> * tests/check-tests-in-builddir.test: Likewise.
> * tests/check-tests_environment.test: Likewise.
> * tests/Makefile.am (TESTS): Update.

> --- /dev/null
> +++ b/tests/check-exported-srcdir.test

> +# Check that the "Simple Tests" driver (either with or without the

Since this test source will be used for both drivers, why not
  s/"Simple Tests"/testsuite/

> +# parallel-tests option enabled) exports the `srcdir' value in the
> +# environment of the tests.  This is documented in the manual.

> +mkdir SrcDir BuildDir

Please let's avoid CamelCase in the future.  We've managed almost two
decades without it, we don't need it now.  (Plus, it doesn't fit the
GCS style.)

> --- /dev/null
> +++ b/tests/check-tests-in-builddir.test

> +# Check that the "Simple Tests" driver can find test in the srcdir as
> +# well as in builddir, and that is prefers those in the builddir.

See above.

> +. ./defs || Exit 1
> +
> +set -e
> +
> +cat >> configure.in << 'END'
> +AC_OUTPUT
> +END
> +
> +cat > Makefile.am << 'END'
> +TESTS = foo.test bar.test
> +EXTRA_DIST = $(TESTS)
> +END
> +
> +cat > foo.test << 'END'
> +#! /bin/sh
> +exit ${FOO_EXIT_STATUS-0}
> +END
> +chmod a+x foo.test
> +
> +$ACLOCAL
> +$AUTOCONF
> +$AUTOMAKE -a
> +
> +mkdir build
> +cd build
> +
> +../configure
> +
> +cat > bar.test << 'END'
> +#! /bin/sh
> +exit 0
> +END
> +chmod a+x bar.test
> +

unset FOO_EXIT_STATUS || :

> +$MAKE check >out 2>&1 || { cat out; Exit1; }
> +cat out
> +grep '\.\./foo' out && Exit 1
> +grep '^PASS: foo.test *$' out
> +grep '^PASS: bar.test *$' out
> +
> +rm -f test-suite.log foo.log bar.log
> +
> +FOO_EXIT_STATUS=1 $MAKE check >out 2>&1 && { cat out; Exit1; }
> +cat out
> +grep '\.\./foo' out && Exit 1
> +grep '^FAIL: foo.test *$' out
> +grep '^PASS: bar.test *$' out
> +
> +rm -f test-suite.log foo.log bar.log
> +
> +# Check that if the same test is present in srcdir and builddir,
> +# the one in builddir is preferred.
> +cp bar.test foo.test
> +FOO_EXIT_STATUS=1 $MAKE check >out 2>&1 || { cat out; Exit1; }
> +cat out
> +grep '^PASS: foo.test *$' out
> +grep '^PASS: bar.test *$' out
> +
> +# The tests in the builddir must be preferred also by "make dist".
> +FOO_EXIT_STATUS=1 $MAKE distcheck

> --- /dev/null
> +++ b/tests/check-tests_environment.test

> +# "Simple Tests" testsuite driver: check TESTS_ENVIRONMENT support.
> +
> +. ./defs || Exit 1
> +
> +set -e
> +
> +cat >> configure.in << 'END'
> +AC_OUTPUT
> +END
> +
> +cat > Makefile.am << 'END'
> +TESTS = foo.test
> +EXTRA_DIST = $(TESTS)
> +END
> +
> +cat > foo.test << 'END'
> +#! /bin/sh
> +test x"$FOO" = x"ok"
> +END
> +chmod a+x foo.test
> +
> +$ACLOCAL
> +$AUTOCONF
> +$AUTOMAKE -a
> +
> +./configure
> +
> +FOO=bad TESTS_ENVIRONMENT='FOO=ok'  $MAKE check
> +FOO=ok  TESTS_ENVIRONMENT='FOO=bad' $MAKE check && Exit 1

I wonder whether we need to unset TESTS_ENVIRONMENT in tests/defs.

> --- /dev/null
> +++ b/tests/parallel-tests-subdir.test

> +# Check that the parallel-tests driver creates parent directories for
> +# the log files when needed.

I think we have a test for this already.  Just
  grep 'TESTS.*/' tests/*.test

(I can understand your desire for more descriptive test names, but not
in easy cases like this one.  No matter how descriptive, you will still
need to search existing tests now and then, simply because the naming
may not be unique.)

> +parallel_tests=yes
> +. ./defs || Exit 1
> +
> +set -e
> +
> +cat >> configure.in << 'END'
> +AC_OUTPUT
> +END
> +
> +cat > Makefile.am << 'END'
> +TESTS = dir1/foo.test dir2/dir3/foo.test
> +TEST_LOG_COMPILER = sh
> +END
> +
> +mkdir dir1 dir2 dir2/dir3
> +echo : > dir1/foo.test
> +echo : > dir2/dir3/foo.test
> +
> +$ACLOCAL
> +$AUTOCONF
> +$AUTOMAKE -a
> +
> +mkdir build
> +cd build
> +../configure
> +$MAKE check
> +find . # For debugging.
> +test -f test-suite.log
> +test -f dir1/foo.log
> +test -f dir2/dir3/foo.log

Thanks,
Ralf