[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[FYI] {GSoC} testsuite: scaffolding to allow use of TAP in our own tests
|
From: |
Stefano Lattarini |
|
Subject: |
[FYI] {GSoC} testsuite: scaffolding to allow use of TAP in our own tests |
|
Date: |
Wed, 3 Aug 2011 12:53:06 +0200 |
|
User-agent: |
KMail/1.13.3 (Linux/2.6.30-2-686; KDE/4.4.4; i686; ; ) |
Now that Automake has initial support for the TAP test protocol,
we can start "eating our own dog food" and rewrite some tests to
use TAP; this should allow for better granularity, especially in
the control of skips and expected failures. With this change, we
set up the initial scaffolding required by the planned TAP tests,
and convert few older tests to use TAP, mostly in order to verify
that there are no obvious errors. The tests in our testsuite that
use the TAP protocol will be marked by the new `.tap' extension.
* tests/plain-functions.sh: New file containing definition of
helper shell functions used by the "protocol-less" tests of the
Automake testsuite.
* tests/tap-functions.sh: New file containing definition of
helper shell functions used by the TAP-producing tests of the
Automake testsuite.
* tests/defs: Remove definitions of some functions that are
now defined in `plain-functions.sh'. Instead, source one of
`plain-functions.sh' or `tap-functions.sh', depending on whether
`$use_tap' is set to "no" or "yes".
(exit trap): Call `late_plan_' if `$use_tap' is true. Also,
unset shell traces before issuing the latest commands, to avoid
confusing the tap driver with spurious output.
* tests/defs-static.in ($use_tap): New variable, by default set
to "yes" if the calling test script has a `.tap' suffix, and to
"no" otherwise. The individual scripts can override it though.
In code sanity-checking the environment, verify that `$use_tap'
is not exported.
* tests/self-check-env-sanitize.test: Update, and small related
reformatting.
* tests/self-check-tap.test: New very minimal self test.
* tests/acoutbs.test, tests/acoutbs2.test, tests/acoutnoq.test,
tests/acoutpt.test, tests/acoutpt2.test, tests/acoutqnl.test:
Removed, merged into ...
* tests/ac-output-old.tap: ... this new test, that uses TAP.
* tests/Makefile.am (TAP_LOG_DRIVER): Define to invoke our own
`tap-driver' script.
(TAP_LOG_DRIVER_FLAGS): Define to `--merge', so that the stdout
and stderr of the test scripts remains synced.
(TEST_EXTENSIONS): Define, to add the `.tap' suffix; also list
the `.test' suffix explicitly.
(EXTRA_DIST): Distribute the new files `plain-functions.sh' and
`tap-functions.sh'.
($(TEST_LOGS)): Depends on them.
(AM_TESTS_ENVIRONMENT): Ensure that `use_tap' is not exported in
the environment of the tests.
(TESTS): Update.
---
ChangeLog | 49 ++++++++
tests/Makefile.am | 19 ++-
tests/Makefile.in | 31 +++--
tests/ac-output-old.tap | 179 +++++++++++++++++++++++++++
tests/acoutbs.test | 42 -------
tests/acoutbs2.test | 47 -------
tests/acoutnoq.test | 35 ------
tests/acoutpt.test | 32 -----
tests/acoutpt2.test | 44 -------
tests/acoutqnl.test | 32 -----
tests/defs | 29 +++--
tests/defs-static.in | 11 ++-
tests/plain-functions.sh | 39 ++++++
tests/self-check-env-sanitize.test | 14 ++-
tests/self-check-tap.test | 34 +++++
tests/tap-functions.sh | 239 ++++++++++++++++++++++++++++++++++++
16 files changed, 611 insertions(+), 265 deletions(-)
create mode 100755 tests/ac-output-old.tap
delete mode 100755 tests/acoutbs.test
delete mode 100755 tests/acoutbs2.test
delete mode 100755 tests/acoutnoq.test
delete mode 100755 tests/acoutpt.test
delete mode 100755 tests/acoutpt2.test
delete mode 100755 tests/acoutqnl.test
create mode 100644 tests/plain-functions.sh
create mode 100755 tests/self-check-tap.test
create mode 100644 tests/tap-functions.sh
diff --git a/ChangeLog b/ChangeLog
index b681294..b9d97c6 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,5 +1,54 @@
2011-08-03 Stefano Lattarini <address@hidden>
+ testsuite: scaffolding to allow use of TAP in our own tests
+ Now that Automake has initial support for the TAP test protocol,
+ we can start "eating our own dog food" and rewrite some tests to
+ use TAP; this should allow for better granularity, especially in
+ the control of skips and expected failures. With this change, we
+ set up the initial scaffolding required by the planned TAP tests,
+ and convert few older tests to use TAP, mostly in order to verify
+ that there are no obvious errors. The tests in our testsuite that
+ use the TAP protocol will be marked by the new `.tap' extension.
+ * tests/plain-functions.sh: New file containing definition of
+ helper shell functions used by the "protocol-less" tests of the
+ Automake testsuite.
+ * tests/tap-functions.sh: New file containing definition of
+ helper shell functions used by the TAP-producing tests of the
+ Automake testsuite.
+ * tests/defs: Remove definitions of some functions that are
+ now defined in `plain-functions.sh'. Instead, source one of
+ `plain-functions.sh' or `tap-functions.sh', depending on whether
+ `$use_tap' is set to "no" or "yes".
+ (exit trap): Call `late_plan_' if `$use_tap' is true. Also,
+ unset shell traces before issuing the latest commands, to avoid
+ confusing the tap driver with spurious output.
+ * tests/defs-static.in ($use_tap): New variable, by default set
+ to "yes" if the calling test script has a `.tap' suffix, and to
+ "no" otherwise. The individual scripts can override it though.
+ In code sanity-checking the environment, verify that `$use_tap'
+ is not exported.
+ * tests/self-check-env-sanitize.test: Update, and small related
+ reformatting.
+ * tests/self-check-tap.test: New very minimal self test.
+ * tests/acoutbs.test, tests/acoutbs2.test, tests/acoutnoq.test,
+ tests/acoutpt.test, tests/acoutpt2.test, tests/acoutqnl.test:
+ Removed, merged into ...
+ * tests/ac-output-old.tap: ... this new test, that uses TAP.
+ * tests/Makefile.am (TAP_LOG_DRIVER): Define to invoke our own
+ `tap-driver' script.
+ (TAP_LOG_DRIVER_FLAGS): Define to `--merge', so that the stdout
+ and stderr of the test scripts remains synced.
+ (TEST_EXTENSIONS): Define, to add the `.tap' suffix; also list
+ the `.test' suffix explicitly.
+ (EXTRA_DIST): Distribute the new files `plain-functions.sh' and
+ `tap-functions.sh'.
+ ($(TEST_LOGS)): Depends on them.
+ (AM_TESTS_ENVIRONMENT): Ensure that `use_tap' is not exported in
+ the environment of the tests.
+ (TESTS): Update.
+
+2011-08-03 Stefano Lattarini <address@hidden>
+
* NEWS: Fix typo, and related reformatting.
2011-08-03 Stefano Lattarini <address@hidden>
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 02d9f94..3e9267b 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -16,6 +16,10 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
+TEST_EXTENSIONS = .test .tap
+TAP_LOG_DRIVER = $(PERL) $(top_srcdir)/lib/tap-driver
+TAP_LOG_DRIVER_FLAGS = --merge
+
MAINTAINERCLEANFILES =
EXTRA_DIST = ChangeLog-old
@@ -107,6 +111,7 @@ XFAIL_TESTS += $(instspc_xfail_tests)
AM_TESTS_ENVIRONMENT = \
test x"$$me" = x || unset me; \
test x"$$required" = x || unset required; \
+ test x"$$use_tap" = x || unset use_tap; \
test x"$$parallel_tests" = x || unset parallel_tests; \
test x"$$original_AUTOMAKE" = x || unset original_AUTOMAKE; \
test x"$$original_ACLOCAL" = x || unset original_ACLOCAL;
@@ -134,12 +139,7 @@ acloca20.test \
acloca21.test \
acloca22.test \
acloca23.test \
-acoutnoq.test \
-acoutpt.test \
-acoutpt2.test \
-acoutqnl.test \
-acoutbs.test \
-acoutbs2.test \
+ac-output-old.tap \
acsilent.test \
acsubst.test \
acsubst2.test \
@@ -888,6 +888,7 @@ self-check-is_newest.test \
self-check-me.test \
self-check-report.test \
self-check-sanity.test \
+self-check-tap.test \
self-check-unindent.test \
sanity.test \
scripts.test \
@@ -1110,6 +1111,9 @@ $(parallel_tests)
EXTRA_DIST += $(TESTS)
+## Files containing auxiliary functions used by our test cases.
+EXTRA_DIST += tap-functions.sh plain-functions.sh
+
test-driver-custom-multitest.log: trivial-test-driver
test-driver-custom-multitest-recheck.log: trivial-test-driver
test-driver-custom-multitest-recheck2.log: trivial-test-driver
@@ -1200,6 +1204,9 @@ EXTRA_DIST += tap-summary-aux.sh
# Dependencies valid for each test case.
$(TEST_LOGS): defs defs-static aclocal-$(APIVERSION) automake-$(APIVERSION)
+# FIXME: this should be made more granular once we have a cleaner
+# subdivision of the tests.
+$(TEST_LOGS): plain-functions.sh tap-functions.sh
clean-local: clean-local-check
.PHONY: clean-local-check
diff --git a/tests/Makefile.in b/tests/Makefile.in
index fa06c4e..8f7809d 100644
--- a/tests/Makefile.in
+++ b/tests/Makefile.in
@@ -171,12 +171,14 @@ RECHECK_LOGS = $(TEST_LOGS)
AM_RECURSIVE_TARGETS = check check-html recheck recheck-html
TEST_SUITE_LOG = test-suite.log
TEST_SUITE_HTML = $(TEST_SUITE_LOG:.log=.html)
-TEST_EXTENSIONS = .test
am__test_logs1 = $(TESTS:=.log)
-TEST_LOGS = $(am__test_logs1:.test.log=.log)
+am__test_logs2 = $(am__test_logs1:.test.log=.log)
TEST_LOG_DRIVER = $(SHELL) $(top_srcdir)/lib/test-driver
TEST_LOG_COMPILE = $(TEST_LOG_COMPILER) $(AM_TEST_LOG_FLAGS) \
$(TEST_LOG_FLAGS)
+TEST_LOGS = $(am__test_logs2:.tap.log=.log)
+TAP_LOG_COMPILE = $(TAP_LOG_COMPILER) $(AM_TAP_LOG_FLAGS) \
+ $(TAP_LOG_FLAGS)
TEST_LOGS_TMP = $(TEST_LOGS:.log=.log-t)
DISTFILES = $(DIST_COMMON) $(DIST_SOURCES) $(TEXINFOS) $(EXTRA_DIST)
ACLOCAL = @ACLOCAL@
@@ -273,9 +275,13 @@ target_alias = @target_alias@
top_build_prefix = @top_build_prefix@
top_builddir = @top_builddir@
top_srcdir = @top_srcdir@
+TEST_EXTENSIONS = .test .tap
+TAP_LOG_DRIVER = $(PERL) $(top_srcdir)/lib/tap-driver
+TAP_LOG_DRIVER_FLAGS = --merge
MAINTAINERCLEANFILES = $(parallel_tests) $(instspc_tests)
EXTRA_DIST = ChangeLog-old gen-parallel-tests instspc-tests.sh \
- $(TESTS) trivial-test-driver testsuite-summary-checks.sh \
+ $(TESTS) tap-functions.sh plain-functions.sh \
+ trivial-test-driver testsuite-summary-checks.sh \
extract-testsuite-summary tap-setup.sh tap-summary-aux.sh
XFAIL_TESTS = all.test auxdir2.test cond17.test gcj6.test \
override-conditional-2.test pr8365-remake-timing.test \
@@ -358,6 +364,7 @@ instspc_xfail_tests = instspc-squote-build.test \
AM_TESTS_ENVIRONMENT = \
test x"$$me" = x || unset me; \
test x"$$required" = x || unset required; \
+ test x"$$use_tap" = x || unset use_tap; \
test x"$$parallel_tests" = x || unset parallel_tests; \
test x"$$original_AUTOMAKE" = x || unset original_AUTOMAKE; \
test x"$$original_ACLOCAL" = x || unset original_ACLOCAL;
@@ -385,12 +392,7 @@ acloca20.test \
acloca21.test \
acloca22.test \
acloca23.test \
-acoutnoq.test \
-acoutpt.test \
-acoutpt2.test \
-acoutqnl.test \
-acoutbs.test \
-acoutbs2.test \
+ac-output-old.tap \
acsilent.test \
acsubst.test \
acsubst2.test \
@@ -1139,6 +1141,7 @@ self-check-is_newest.test \
self-check-me.test \
self-check-report.test \
self-check-sanity.test \
+self-check-tap.test \
self-check-unindent.test \
sanity.test \
scripts.test \
@@ -1429,7 +1432,7 @@ tap-summary-color.test
all: all-am
.SUFFIXES:
-.SUFFIXES: .html .log .test .trs
+.SUFFIXES: .html .log .tap .test .trs
$(srcdir)/Makefile.in: $(srcdir)/Makefile.am $(srcdir)/parallel-tests.am
$(srcdir)/instspc-tests.am $(am__configure_deps)
@for dep in $?; do \
case '$(am__configure_deps)' in \
@@ -1639,6 +1642,11 @@ recheck recheck-html:
--log-file '$*.log' --trs-file '$*.trs' \
$(am__common_driver_flags) $(AM_TEST_LOG_DRIVER_FLAGS)
$(TEST_LOG_DRIVER_FLAGS) -- $(TEST_LOG_COMPILE) "$$tst" \
$(AM_TESTS_FD_REDIRECT)
+.tap.log:
+ @p='$<'; $(am__check_pre) $(TAP_LOG_DRIVER) --test-name "$$f" \
+ --log-file '$*.log' --trs-file '$*.trs' \
+ $(am__common_driver_flags) $(AM_TAP_LOG_DRIVER_FLAGS)
$(TAP_LOG_DRIVER_FLAGS) -- $(TAP_LOG_COMPILE) "$$tst" \
+ $(AM_TESTS_FD_REDIRECT)
distdir: $(DISTFILES)
@srcdirstrip=`echo "$(srcdir)" | sed 's/[].[^$$\\*]/\\\\&/g'`; \
@@ -1891,6 +1899,9 @@ tap-summary.log tap-summary-color.log: tap-summary-aux.sh
# Dependencies valid for each test case.
$(TEST_LOGS): defs defs-static aclocal-$(APIVERSION) automake-$(APIVERSION)
+# FIXME: this should be made more granular once we have a cleaner
+# subdivision of the tests.
+$(TEST_LOGS): plain-functions.sh tap-functions.sh
clean-local: clean-local-check
.PHONY: clean-local-check
diff --git a/tests/ac-output-old.tap b/tests/ac-output-old.tap
new file mode 100755
index 0000000..74bbba3
--- /dev/null
+++ b/tests/ac-output-old.tap
@@ -0,0 +1,179 @@
+#!/bin/sh
+# Copyright (C) 1996, 2000, 2001, 2002, 2010, 2011 Free Software
+# Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Test for various older bugs related to quoting, escaping and
+# line breaking in the use of AC_OUTPUT. Synthesised by a bunch
+# of older tests (referenced below).
+
+. ./defs || Exit 1
+
+rm -f configure.in depcomp # Not required.
+
+# -----------------------------------------------------------------------
+
+# Test for bug reported by François Pinard.
+# If \ is in AC_OUTPUT, automake barfs.
+# This was the old test `acoutbs.test'.
+
+# Also test for bug reported by David A. Swierczek.
+# Another bug with \ in AC_OUTPUT (this time with whitespace).
+# This was the old test `acoutbs2.test'.
+
+acoutbs_check ()
+{
+ dir=acoutbs$1
+ mkdir $dir
+ cd $dir
+ cat > configure.in
+ touch Makefile.am zot.in
+ command_ok_ "aclocal groks '\\' in AC_OUTPUT ($dir)" $ACLOCAL
+ command_ok_ "automake groks '\\' in AC_OUTPUT ($dir)" $AUTOMAKE
+ command_ok_ "autoconf groks '\\' in AC_OUTPUT ($dir)" $AUTOCONF
+ command_ok_ "can ./configure in $dir" ./configure
+ command_ok_ "zot created in $dir" test -f zot
+ ls -a > lst || bailout_ "cannot get file listing in $dir"
+ command_not_ok_ "'\\' not leaked in filenames in $dir" grep '\\' lst
+ cd ..
+}
+
+acoutbs_check 1 << 'END'
+AC_INIT
+AM_INIT_AUTOMAKE(nonesuch, nonesuch)
+AC_OUTPUT(Makefile \
+zot)
+END
+
+acoutbs_check 2 << 'END'
+AC_INIT
+AM_INIT_AUTOMAKE(nonesuch, nonesuch)
+AC_ARG_PROGRAM
+AC_PROG_INSTALL
+AC_PROG_MAKE_SET
+AC_OUTPUT(\
+ Makefile \
+ zot
+)
+END
+
+# -----------------------------------------------------------------------
+
+# Test for bug reported by Jerome Santini.
+# If I put this line in my configure.in:
+# AC_OUTPUT(Makefile src/Makefile tests/Makefile, echo timestamp > stamp-h)dnl
+# automake is not happy:
+# [ ... ]
+# This was the old test `acoutnoq.test'.
+
+mkdir acoutnoq
+cd acoutnoq
+
+cat > configure.in << 'END'
+AC_INIT
+AM_INIT_AUTOMAKE(nonesuch, nonesuch)
+AC_OUTPUT(Makefile, [true])
+END
+
+: > Makefile.am
+
+command_ok_ "aclocal and quoted AC_OUTPUT second argument" $ACLOCAL
+command_ok_ "automake and quoted AC_OUTPUT second argument" $AUTOMAKE
+
+cd ..
+
+# -----------------------------------------------------------------------
+
+# Test for bug when AC_OUTPUT has 2 args on the same line, eg:
+# AC_OUTPUT([Makefile automake tests/Makefile],[chmod +x automake])
+# This was the old test `acoutpt.test'.
+
+mkdir acoutpt
+cd acoutpt
+
+cat > configure.in << 'END'
+AC_INIT
+AM_INIT_AUTOMAKE(nonesuch, nonesuch)
+AC_OUTPUT([Makefile], [true])
+END
+
+: > Makefile.am
+
+command_ok_ "aclocal and two AC_OUTPUT arguments on same line" $ACLOCAL
+command_ok_ "automake and two AC_OUTPUT arguments on same line" $AUTOMAKE
+
+cd ..
+
+# -----------------------------------------------------------------------
+
+# Test for bug reported by Eric Magnien.
+# This was the old test `acoutpt2.test'.
+
+mkdir acoutpt2
+cd acoutpt2
+
+# Name of the current "subtest".
+cur=acoutput2
+
+cat > configure.in <<END
+AC_INIT([$cur], [1.0])
+AM_INIT_AUTOMAKE
+AC_OUTPUT([subdir/Makefile subdir/foo Makefile foo], [true])
+END
+
+mkdir subdir
+
+: > Makefile.am
+: > subdir/Makefile.am
+: > foo.in
+: > subdir/foo.in
+
+command_ok_ "aclocal and AC_OUTPUT ($cur)" $ACLOCAL
+command_ok_ "automake and AC_OUTPUT ($cur)" $AUTOMAKE
+command_ok_ "foo.in mentioned two times in Makefile.in ($cur)" \
+ test `$FGREP foo.in Makefile.in | wc -l` -eq 2
+
+# This ought to work as well.
+command_ok_ "'automake -a -f' and AC_OUTPUT ($cur)" \
+ $AUTOMAKE --add-missing --force-missing
+
+cd ..
+
+# -----------------------------------------------------------------------
+
+# Test for bug reported by François Pinard.
+# This was the old test `acoutqnl.test'.
+
+mkdir acoutqnl
+cd acoutqnl
+
+cat > configure.in << 'END'
+AC_INIT
+AM_INIT_AUTOMAKE(nonesuch, nonesuch)
+AC_OUTPUT([Makefile],
+[echo zardoz has spoken])
+END
+
+: > Makefile.am
+
+command_ok_ "aclocal and two AC_OUTPUT arguments on two lines" $ACLOCAL
+command_ok_ "automake and two AC_OUTPUT arguments on two lines" $AUTOMAKE
+
+cd ..
+
+# -----------------------------------------------------------------------
+
+# And we're done.
+Exit 0
diff --git a/tests/acoutbs.test b/tests/acoutbs.test
deleted file mode 100755
index 232267d..0000000
--- a/tests/acoutbs.test
+++ /dev/null
@@ -1,42 +0,0 @@
-#!/bin/sh
-# Copyright (C) 1996, 2000, 2001, 2002, 2010, 2011 Free Software
-# Foundation, Inc.
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; either version 2, or (at your option)
-# any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program. If not, see <http://www.gnu.org/licenses/>.
-
-# Test for bug reported by François Pinard.
-# If \ is in AC_OUTPUT, automake barfs.
-
-. ./defs || Exit 1
-
-cat > configure.in << 'END'
-AC_INIT
-AM_INIT_AUTOMAKE(nonesuch, nonesuch)
-AC_OUTPUT(Makefile \
-zot)
-END
-
-: > Makefile.am
-: > zot.in
-
-$ACLOCAL
-$AUTOCONF
-$AUTOMAKE
-
-./configure
-
-test -f zot || Exit 1
-ls -a | grep '\\' && Exit 1
-
-:
diff --git a/tests/acoutbs2.test b/tests/acoutbs2.test
deleted file mode 100755
index cd1cd9d..0000000
--- a/tests/acoutbs2.test
+++ /dev/null
@@ -1,47 +0,0 @@
-#! /bin/sh
-# Copyright (C) 1996, 2000, 2002, 2010, 2011 Free Software Foundation,
-# Inc.
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; either version 2, or (at your option)
-# any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program. If not, see <http://www.gnu.org/licenses/>.
-
-# Test for bug reported by David A. Swierczek.
-# Another bug with \ in AC_OUTPUT (this time with whitespace).
-
-. ./defs || Exit 1
-
-cat > configure.in << 'END'
-AC_INIT
-AM_INIT_AUTOMAKE(nonesuch, nonesuch)
-AC_ARG_PROGRAM
-AC_PROG_INSTALL
-AC_PROG_MAKE_SET
-AC_OUTPUT(\
- Makefile \
- zot
-)
-END
-
-: > Makefile.am
-: > zot.in
-
-$ACLOCAL
-$AUTOCONF
-$AUTOMAKE
-
-./configure
-
-test -f zot || Exit 1
-ls -a | grep '\\' && Exit 1
-
-:
diff --git a/tests/acoutnoq.test b/tests/acoutnoq.test
deleted file mode 100755
index d49f878..0000000
--- a/tests/acoutnoq.test
+++ /dev/null
@@ -1,35 +0,0 @@
-#!/bin/sh
-# Copyright (C) 1996, 2000, 2001, 2002, 2010, 2011 Free Software
-# Foundation, Inc.
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; either version 2, or (at your option)
-# any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program. If not, see <http://www.gnu.org/licenses/>.
-
-# Test for bug reported by Jerome Santini.
-# If I put this line in my configure.in:
-# AC_OUTPUT(Makefile src/Makefile tests/Makefile, echo timestamp > stamp-h)dnl
-# automake is not happy:
-# [ ... ]
-
-. ./defs || Exit 1
-
-cat > configure.in << 'END'
-AC_INIT
-AM_INIT_AUTOMAKE(nonesuch, nonesuch)
-AC_OUTPUT(Makefile, [true])
-END
-
-: > Makefile.am
-
-$ACLOCAL
-$AUTOMAKE
diff --git a/tests/acoutpt.test b/tests/acoutpt.test
deleted file mode 100755
index 12b6427..0000000
--- a/tests/acoutpt.test
+++ /dev/null
@@ -1,32 +0,0 @@
-#!/bin/sh
-# Copyright (C) 1996, 2000, 2001, 2002, 2010, 2011 Free Software
-# Foundation, Inc.
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; either version 2, or (at your option)
-# any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program. If not, see <http://www.gnu.org/licenses/>.
-
-# Test for bug when AC_OUTPUT has 2 args on the same line, eg:
-# AC_OUTPUT([Makefile automake tests/Makefile],[chmod +x automake])
-
-. ./defs || Exit 1
-
-cat > configure.in << 'END'
-AC_INIT
-AM_INIT_AUTOMAKE(nonesuch, nonesuch)
-AC_OUTPUT([Makefile], [true])
-END
-
-: > Makefile.am
-
-$ACLOCAL
-$AUTOMAKE
diff --git a/tests/acoutpt2.test b/tests/acoutpt2.test
deleted file mode 100755
index d972989..0000000
--- a/tests/acoutpt2.test
+++ /dev/null
@@ -1,44 +0,0 @@
-#! /bin/sh
-# Copyright (C) 2001, 2002, 2010, 2011 Free Software Foundation, Inc.
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; either version 2, or (at your option)
-# any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program. If not, see <http://www.gnu.org/licenses/>.
-
-# Test for bug reported by Eric Magnien.
-
-. ./defs || Exit 1
-
-cat > configure.in <<END
-AC_INIT([$me], [1.0])
-AM_INIT_AUTOMAKE
-AC_OUTPUT([subdir/Makefile subdir/foo Makefile foo], [true])
-END
-
-mkdir subdir
-
-: > Makefile.am
-: > subdir/Makefile.am
-: > foo.in
-: > subdir/foo.in
-
-$ACLOCAL
-$AUTOMAKE
-
-count=`$FGREP foo.in Makefile.in | wc -l`
-test $count -eq 2
-
-# This ought to work as well.
-$ACLOCAL
-$AUTOMAKE --add-missing --force-missing
-
-:
diff --git a/tests/acoutqnl.test b/tests/acoutqnl.test
deleted file mode 100755
index 5cfac28..0000000
--- a/tests/acoutqnl.test
+++ /dev/null
@@ -1,32 +0,0 @@
-#! /bin/sh
-# Copyright (C) 1996, 2000, 2001, 2002, 2010, 2011 Free Software
-# Foundation, Inc.
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; either version 2, or (at your option)
-# any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program. If not, see <http://www.gnu.org/licenses/>.
-
-# Test for bug reported by François Pinard.
-
-. ./defs || Exit 1
-
-cat > configure.in << 'END'
-AC_INIT
-AM_INIT_AUTOMAKE(nonesuch, nonesuch)
-AC_OUTPUT([Makefile],
-[echo zardoz has spoken])
-END
-
-: > Makefile.am
-
-$ACLOCAL
-$AUTOMAKE
diff --git a/tests/defs b/tests/defs
index d3f6de0..2dca4cf 100644
--- a/tests/defs
+++ b/tests/defs
@@ -117,20 +117,19 @@ Exit ()
exit $1
}
-# Print warnings (e.g., about skipped and failed tests) to this file
-# number. Override by putting, say:
-# stderr_fileno_=9; export stderr_fileno_; exec 9>&2;
-# in the definition of AM_TESTS_ENVIRONMENT.
-# This is useful when using automake's parallel tests mode, to print the
-# reason for skip/failure to console, rather than to the *.log files.
-: ${stderr_fileno_=2}
-
-# Copied from Gnulib's `tests/init.sh'.
-warn_ () { echo "$@" 1>&$stderr_fileno_; }
-fail_ () { warn_ "$me: failed test: $@"; Exit 1; }
-skip_ () { warn_ "$me: skipped test: $@"; Exit 77; }
-fatal_ () { warn_ "$me: hard error: $@"; Exit 99; }
-framework_failure_ () { warn_ "$me: set-up failure: $@"; Exit 99; }
+if test $using_tap = yes; then
+ funcs_file_=tap-functions.sh
+else
+ funcs_file_=plain-functions.sh
+fi
+
+if test -f "$testsrcdir/$funcs_file_"; then
+ . "$testsrcdir/$funcs_file_"
+else
+ echo "$me: $testsrcdir/$funcs_file_ not found, check \$testsrcdir" >&2
+ Exit 99
+fi
+unset funcs_file_
# cross_compiling
# ---------------
@@ -641,8 +640,10 @@ if test "$sh_errexit_works" = yes; then
rm -rf $testSubDir
;;
esac
+ set +x
test "$signal" != 0 && echo "$me: caught signal $signal"
echo "$me: exit $exit_status"
+ test $using_tap = yes && late_plan_
exit $exit_status
' 0
for signal in 1 2 13 15; do
diff --git a/tests/defs-static.in b/tests/defs-static.in
index 6d82956..f1294c6 100644
--- a/tests/defs-static.in
+++ b/tests/defs-static.in
@@ -66,7 +66,9 @@ fi
# Check that the environment is properly sanitized.
# Having variables exported to the empty string is OK, since our code
# treats such variables as if they were unset.
-for var in me required parallel_tests original_AUTOMAKE original_ACLOCAL; do
+for var in me required use_tap parallel_tests \
+ original_AUTOMAKE original_ACLOCAL;
+do
if eval "test x\"\$$var\" != x" && env | grep "^$var=" >/dev/null; then
echo "$argv0: variable \`$var' is set in the environment:" \
"this is unsafe" >&2
@@ -75,6 +77,13 @@ for var in me required parallel_tests original_AUTOMAKE
original_ACLOCAL; do
done
unset var
+# See whether the current test script is expected to use TAP or not.
+# Use a sensible default, while allowing the scripts to override this
+# check.
+if test -z "$using_tap"; then
+ case $argv0 in *.tap) using_tap=yes;; *) using_tap=no;; esac
+fi
+
testsrcdir='@abs_srcdir@'
top_testsrcdir='@abs_top_srcdir@'
testbuilddir='@abs_builddir@'
diff --git a/tests/plain-functions.sh b/tests/plain-functions.sh
new file mode 100644
index 0000000..c142f05
--- /dev/null
+++ b/tests/plain-functions.sh
@@ -0,0 +1,39 @@
+# -*- shell-script -*-
+#
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Helper functions used by "plain" tests of the Automake testsuite
+# (i.e., tests that don't use any test protocol).
+
+# Print warnings (e.g., about skipped and failed tests) to this file
+# number. Override by putting, say:
+# stderr_fileno_=9; export stderr_fileno_; exec 9>&2;
+# in the definition of AM_TESTS_ENVIRONMENT.
+# This is useful when using automake's parallel tests mode, to print the
+# reason for skip/failure to console, rather than to the *.log files.
+: ${stderr_fileno_=2}
+
+# Copied from Gnulib's `tests/init.sh'.
+warn_ () { echo "$@" 1>&$stderr_fileno_; }
+fail_ () { warn_ "$me: failed test: $@"; Exit 1; }
+skip_ () { warn_ "$me: skipped test: $@"; Exit 77; }
+fatal_ () { warn_ "$me: hard error: $@"; Exit 99; }
+framework_failure_ () { warn_ "$me: set-up failure: $@"; Exit 99; }
+
+# For compatibility with TAP functions.
+skip_all_ () { skip "$@"; }
+
+:
diff --git a/tests/self-check-env-sanitize.test
b/tests/self-check-env-sanitize.test
index 169f003..1f7dba2 100755
--- a/tests/self-check-env-sanitize.test
+++ b/tests/self-check-env-sanitize.test
@@ -23,10 +23,20 @@
set -x
exec 5>&1
-for var in me parallel_tests required original_AUTOMAKE original_ACLOCAL; do
+vars='
+ me
+ parallel_tests
+ required
+ use_tap
+ original_AUTOMAKE
+ original_ACLOCAL
+'
+
+for var in $vars; do
env "$var=foo" $SHELL -c '. ./defs' foo.test && exit 1
env "$var=foo" $SHELL -c '. ./defs' foo.test 2>&1 1>&5 \
- | grep "foo\.test:.* variable \`$var'.* in the environment.*unsafe" ||
exit 1
+ | grep "foo\.test:.* variable \`$var'.* in the environment.*unsafe" \
+ || exit 1
done
:
diff --git a/tests/self-check-tap.test b/tests/self-check-tap.test
new file mode 100755
index 0000000..43a8bac
--- /dev/null
+++ b/tests/self-check-tap.test
@@ -0,0 +1,34 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Sanity check for the automake testsuite.
+# Make sure that $using_tap gets automatically defined by `./defs-static',
+# but can be overridden by the individual tests.
+
+. ./defs-static || exit 1
+
+set -ex
+
+$SHELL -c '. ./defs-static && test $using_tap = yes' foo.tap
+$SHELL -c '. ./defs-static && test $using_tap = no' foo.test
+$SHELL -c '. ./defs-static && test $using_tap = no' tap
+$SHELL -c '. ./defs-static && test $using_tap = no' tap.test
+$SHELL -c '. ./defs-static && test $using_tap = no' foo-tap
+
+$SHELL -c 'using_tap=no; . ./defs-static; test $using_tap = no' foo.tap
+$SHELL -c 'using_tap=yes; . ./defs-static; test $using_tap = yes' foo.test
+
+:
diff --git a/tests/tap-functions.sh b/tests/tap-functions.sh
new file mode 100644
index 0000000..0104d39
--- /dev/null
+++ b/tests/tap-functions.sh
@@ -0,0 +1,239 @@
+# -*- shell-script -*-
+#
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Helper functions used by TAP-producing tests of the Automake testsuite.
+
+#
+# IMPORTANT: All the functions defined in this file can *not* be used
+# from within a subshell, unless explicitly noted otherwise.
+#
+
+# The count of the TAP test results seen so far.
+tap_count_=0
+
+# The first "test -n" tries to avoid extra forks when possible.
+if test -n "${ZSH_VERSION}${BASH_VERSION}" \
+ || (eval 'test $((1 + 1)) = 2') >/dev/null 2>&1
+then
+ # Use of 'eval' needed to protect dumber shells from parsing errors.
+ eval 'incr_tap_count_ () { tap_count_=$(($tap_count_ + 1)); }'
+else
+ incr_tap_count_ () { tap_count_=`expr $tap_count_ + 1`; }
+fi
+
+# plan_ NUMBER-OF-PLANNED-TESTS
+# -----------------------------
+# Print a TAP plan for the given number of tests. This must be called
+# before reporting any test result; in fact, it must be called before
+# emitting anything on standard output.
+plan_ ()
+{
+ echo "1..$1"
+ have_tap_plan_=yes
+}
+
+# late_plan_
+# ----------
+# Print a TAP plan that accounts for the number of tests seen so far.
+# This must be called after all the tests result have been reported;
+# in fact, after this has been called, nothing more can be print on
+# standard output.
+late_plan_ ()
+{
+ echo "1..$tap_count_"
+ have_tap_plan_=yes
+}
+
+# Initialize it to avoid interferences from the environment.
+have_tap_plan_=no
+
+# diag_ [EXPLANATION]
+# ------------------
+# Report the given text as TAP diagnostic.
+diag_ ()
+{
+ test $# -eq 0 || echo "# $*"
+}
+
+# warn_ [EXPLANATION]
+# ------------------
+# Give a warning (using TAP diagnostic).
+warn_ ()
+{
+ diag_ "WARNING:" ${1-"(unknown warning)"} ${1+"$@"}
+}
+
+# result_ RESULT [DESCRIPTION...]
+# -------------------------------
+# Report a test case with the given result.
+result_ ()
+{
+ incr_tap_count_
+ tap_result_=$1; shift
+ echo "$tap_result_" $tap_count_ ${1+"- $*"}
+}
+
+# tap_with_directive_ RESULT DIRECTIVE [-r REASON] [DESCRIPTION...]
+# -----------------------------------------------------------------
+# Write a tap result with the given directive (can be "TODO" or "SKIP").
+# The REASON, if given is appended after the directive. This function is
+# for internal use only.
+result_with_directive_ ()
+{
+ incr_tap_count_
+ tap_result_=$1; shift
+ tap_directive_=$1; shift
+ case $1 in -r) tap_reason_=" $2" shift 2;; *) tap_reason_="";; esac
+ echo "$tap_result_" $tap_count_ ${1+"- $*"} \
+ "# ${tap_directive_}${tap_reason_}"
+}
+
+# ok_ [DESCRIPTION...]
+# --------------------
+# Report a successful test.
+ok_ ()
+{
+ result_ 'ok' ${1+"$@"}
+}
+
+# not_ok_ [DESCRIPTION...]
+# ------------------------
+# Report a failed test.
+not_ok_ ()
+{
+ result_ 'not ok' ${1+"$@"}
+}
+
+# skip_ [-r REASON] [DESCRIPTION...]
+# ----------------------------------
+# Report a skipped test. If the `-r' option is present, its argument is
+# give as the reason of the skip.
+skip_ ()
+{
+ result_with_directive_ 'ok' SKIP ${1+"$@"}
+}
+
+# skip_row_ COUNT [-r REASON] [DESCRIPTION...]
+# --------------------------------------------
+# Report a COUNT of skipped test, with the given reason and descriptions
+# (if any). Useful to avoid cascade failures in case a fair number of
+# tests depend on an earlier one that failed.
+skip_row_ ()
+{
+ skip_count_=$1; shift
+ for i_ in `seq $skip_count_`; do skip_ ${1+"$@"}; done
+}
+
+# xfail_ [-r REASON] [DESCRIPTION...]
+# ----------------------------------
+# Report a test that failed expectedly. If the `-r' option is present, its
+# argument is give as the reason why the failure is expected.
+xfail_ ()
+{
+ result_with_directive_ 'not ok' TODO ${1+"$@"}
+}
+
+# xpass_ [-r REASON] [DESCRIPTION...]
+# -----------------------------------
+# Report a test that failed unexpectedly. If the `-r' option is present, its
+# argument is give as the reason why the failure is expected.
+xpass_ ()
+{
+ result_with_directive_ 'ok' TODO ${1+"$@"}
+}
+
+# skip_all_ [REASON ...]
+# ----------------------
+# Skip all the tests in a test script. Must be used before calling `plan_'
+# or reporting any test result. Can't be used from within a subshell.
+skip_all_ ()
+{
+ echo "1..0 # SKIP" ${1+"$@"}
+ Exit 0
+}
+
+# bailout_ [REASON ...]
+# ---------------------
+# Stop the execution of the current test suite right now, due to an
+# unrecoverable error. Can be called at any point, but cannot be used
+# from within a subshell.
+bailout_ ()
+{
+ echo 'Bail out!' ${1+"$@"}
+ Exit 99
+}
+
+# fatal_ [REASON ...]
+# -------------------
+# Same as `bailout_'; for compatibility with `plain-functions.sh'.
+fatal_ ()
+{
+ bailout_ ${1+"$@"}
+}
+
+# framework_failure_ [REASON ...]
+# -------------------------------
+# Stop the execution of the current test suite right now, due to an
+# unrecoverable error in the set-up of the test case. Can be called
+# at any point, but cannot be used from within a subshell.
+framework_failure_ ()
+{
+ bailout_ "set-up failure"${1+": $*"}
+}
+
+# command_ok_ TEST-DESCRIPTION [--] CMD [ARGS...]
+# -----------------------------------------------
+# Report a passed test if the given command returns with success,
+# a failed test otherwise.
+command_ok_ ()
+{
+ tap_desc_=$1; shift
+ test x"$1" != x"--" || shift
+ if "$@"; then
+ ok_ "$tap_desc_"
+ else
+ not_ok_ "$tap_desc_"
+ fi
+}
+
+# command_not_ok_ TEST-DESCRIPTION [--] CMD [ARGS...]
+# ---------------------------------------------------
+# Report a failed test if the given command returns with success,
+# a failed test otherwise.
+command_not_ok_ ()
+{
+ tap_desc_=$1; shift
+ test x"$1" != x"--" || shift
+ if "$@"; then
+ not_ok_ "$tap_desc_"
+ else
+ ok_ "$tap_desc_"
+ fi
+}
+
+
+# reset_test_count_ COUNT
+# -----------------------
+# Reset the count of the TAP test results seen so far to COUNT.
+# This function is for use in corner cases only (e.g., when `ok_' and
+# `not_ok_' must be used inside a subshell). Be careful when using it!
+reset_test_count_ ()
+{
+ tap_count_=$1
+}
+
+:
--
1.7.2.3
| [Prev in Thread] |
Current Thread |
[Next in Thread] |
- [FYI] {GSoC} testsuite: scaffolding to allow use of TAP in our own tests,
Stefano Lattarini <=