Re: [PATCH] {GSoC} docs: improve, extend and fix documentation on TAP support

[Stefano Lattarini]

On Tuesday 02 August 2011, Stefano Lattarini wrote:
> * doc/automake.texi ("Using the TAP test protocol"): Divide this
> section into ...
> ("Introduction to TAP", "Use TAP with the Automake test harness",
> "Incompatibilities with other TAP parsers and drivers", "Links
> and external resources"): ... these subsections, extend them by
> adding more information and examples, and improve them by removing
> incomplete and/or temporary wordings and TODO items.
> ("Script-based Testsuites", "Parallel Test Harness"): Add a couple
> of anchors to improve the granularity of cross-references.
> * tests/tap-doc2.test: New test, verifying the correctness of the
> new examples given in the manual.
> * tests/Makefile.am (tap_other_tests): Add the new test.
> ---
>  ChangeLog           |    9 ++-
>  doc/automake.texi   |  181 ++++++++++++++++++++++++++++++++++++--------------
>  tests/Makefile.am   |    1 +
>  tests/Makefile.in   |    1 +
>  tests/tap-doc2.test |  139 +++++++++++++++++++++++++++++++++++++++
>  5 files changed, 276 insertions(+), 55 deletions(-)
>  create mode 100755 tests/tap-doc2.test
> 
> diff --git a/ChangeLog b/ChangeLog
> index d09b308..6d2a016 100644
> --- a/ChangeLog
> +++ b/ChangeLog
> @@ -6,12 +6,13 @@
>       ("Introduction to TAP", "Use TAP with the Automake test harness",
>       "Incompatibilities with other TAP parsers and drivers", "Links
>       and external resources"): ... these subsections, extend them by
> -     adding more information, and improve them by removing incomplete
> -     and/or temporary wordings and TODO items.  Also ...
> -     ("Links and external resources"): ... add a link to the `tap4j'
> -     project.
> +     adding more information and examples, and improve them by removing
> +     incomplete and/or temporary wordings and TODO items.
>       ("Script-based Testsuites", "Parallel Test Harness"): Add a couple
>       of anchors to improve the granularity of cross-references.
> +     * tests/tap-doc2.test: New test, verifying the correctness of the
> +     new examples given in the manual.
> +     * tests/Makefile.am (tap_other_tests): Add the new test.
>  
> [SNIP]
>
Hmph, the diffs that I've posted were only a squash-in that amended an older
version of the patch I had on my local repository.  I should have attached
the correct patch now.

Thanks, and sorry for the noise,
  Stefano

From bb157eb298ed6f494a15347008d3eaaf2daaa07b Mon Sep 17 00:00:00 2001
Message-Id: <address@hidden>
From: Stefano Lattarini <address@hidden>
Date: Tue, 2 Aug 2011 12:05:20 +0200
Subject: [PATCH] docs: improve, extend and fix documentation on TAP support

* doc/automake.texi ("Using the TAP test protocol"): Divide this
section into ...
("Introduction to TAP", "Use TAP with the Automake test harness",
"Incompatibilities with other TAP parsers and drivers", "Links
and external resources"): ... these subsections, extend them by
adding more information and examples, and improve them by removing
incomplete and/or temporary wordings and TODO items.
("Script-based Testsuites", "Parallel Test Harness"): Add a couple
of anchors to improve the granularity of cross-references.
* tests/tap-doc2.test: New test, verifying the correctness of the
new examples given in the manual.
* tests/Makefile.am (tap_other_tests): Add the new test.
---
 ChangeLog           |   16 ++++
 doc/automake.texi   |  235 +++++++++++++++++++++++++++++++++++++++++----------
 tests/Makefile.am   |    1 +
 tests/Makefile.in   |    1 +
 tests/tap-doc2.test |  139 ++++++++++++++++++++++++++++++
 5 files changed, 347 insertions(+), 45 deletions(-)
 create mode 100755 tests/tap-doc2.test

diff --git a/ChangeLog b/ChangeLog
index 7bc42a5..6d2a016 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,19 @@
+2011-08-02  Stefano Lattarini  <address@hidden>
+
+       docs: improve, extend and fix documentation on TAP support
+       * doc/automake.texi ("Using the TAP test protocol"): Divide this
+       section into ...
+       ("Introduction to TAP", "Use TAP with the Automake test harness",
+       "Incompatibilities with other TAP parsers and drivers", "Links
+       and external resources"): ... these subsections, extend them by
+       adding more information and examples, and improve them by removing
+       incomplete and/or temporary wordings and TODO items.
+       ("Script-based Testsuites", "Parallel Test Harness"): Add a couple
+       of anchors to improve the granularity of cross-references.
+       * tests/tap-doc2.test: New test, verifying the correctness of the
+       new examples given in the manual.
+       * tests/Makefile.am (tap_other_tests): Add the new test.
+
 2011-08-01  Stefano Lattarini  <address@hidden>
 
        testsuite: separate the only failing check of an xfailing test
diff --git a/doc/automake.texi b/doc/automake.texi
index d02e6ca..c1ee0a5 100644
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -330,6 +330,13 @@ Simple Tests
 * Serial Test Harness::         Older (and obsolescent) serial test harness
 * Parallel Test Harness::       Generic concurrent test harness
 
+Using the TAP test protocol
+
+* Introduction to TAP::
+* Use TAP with the Automake test harness::
+* Incompatibilities with other TAP parsers and drivers::
+* Links and external resources::
+
 Custom Test Drivers
 
 * Overview of Custom Test Drivers Support::
@@ -8807,6 +8814,8 @@ might change, or they might even have no effect at all 
(for example,
 in tests using TAP, there is not way to disable hard errors, and the
 @code{DISABLE_HARD_ERRORS} variable has no effect on them).
 
address@hidden progress on console}
address@hidden Testsuite progress on console
 The result of each test case run by the scripts in @code{TESTS} will be
 printed on standard output, along with the test name.  For test protocols
 that allow more test cases per test script (such as TAP), a number,
@@ -8973,6 +8982,7 @@ tests that have not completed in a prior run, summary and 
verbose output in
 This harness is still somewhat experimental and may undergo changes in
 order to satisfy additional portability requirements.
 
address@hidden of test metadata}
 @vindex TEST_SUITE_LOG
 @vindex TESTS
 @cindex @file{.log} files
@@ -9520,23 +9530,188 @@ harness should remain untouched, and continue to work 
correctly.
 @node Using the TAP test protocol
 @section Using the TAP test protocol
 
-Brief introduction to TAP and its philosophy.
address@hidden
+* Introduction to TAP::
+* Use TAP with the Automake test harness::
+* Incompatibilities with other TAP parsers and drivers::
+* Links and external resources::
address@hidden menu
 
-Real-word uses of TAP: testsuite of @command{perl} and of many
-perl modules, and of @command{git}.  Third-party libraries and
-utilities that can generate TAP (tell also to look at links
-below).
address@hidden Introduction to TAP
address@hidden Introduction to TAP
+
+TAP, the Test Anything Protocol, is a simple text-based interface between
+testing modules or programs and a test harness.  The tests (also called
+``TAP producers'' in this context) write test results in a simple format
+on standard output; a test harness (also called ``TAP consumer'') will
+parse and interpret these results, and properly present them to the user,
+and/or register them for later analysis.  The exact details of how this
+is accomplished can vary among different test harnesses.  The Automake
+parallel harness will present the results on the console in the usual
+fashion (@pxref{Testsuite progress on console}), and will use the
address@hidden files (@pxref{Basics of test metadata}) to store the test
+results and related metadata.  Apart from that, it will try to remain
+as much compatible as possible with pre-existing and widespread utilities,
+such as the @uref{http://search.cpan.org/~andya/Test-Harness/bin/prove,
address@hidden utility}, at least for the simpler usages.
+
+TAP started its life as part of the test harness for Perl, but today
+it has been (mostly) standardized, and has various independent
+implementations in different languages; among them, C, C++, Perl,
+Python, PHP, and Java.  For a semi-official specification of the
+TAP protocol, please refer to the documentation of
address@hidden://search.cpan.org/~petdance/Test-Harness/lib/Test/Harness/TAP.pod,
+      @samp{Test::Harness::TAP}}.
+
+The most relevant real-world usages of TAP are obviously in the testsuites
+of @command{perl} and of many perl modules.  Still, also other important
+third-party packages, such as @uref{http://git-scm.com/, @command{git}},
+use TAP in their testsuite.
+
address@hidden Use TAP with the Automake test harness
address@hidden Use TAP with the Automake test harness
+
+Currently, the TAP driver that comes with Automake requires a perl
+interpreter to work, and requires various by-hand steps on the
+developer's part (this should be fixed in future Automake versions).
+You'll have grab the @file{tap-driver} script from the Automake
+distribution by hand, copy it in your source tree, add code to
address@hidden to search a perl interpreter and to define the
address@hidden(PERL)} variable accordingly, and use the Automake support
+for third-party test drivers to instruct the harness to use the
address@hidden to run your TAP-producing tests.  See the example
+below for clarification.
+
+Apart from the options common to all the Automake test drivers
+(@pxref{Command-line arguments for test drivers}), the @file{tap-driver}
+supports the following options, whose names are chosen for enhanced
+compatibility with the @command{prove} utility.
 
address@hidden
-A harness must only read TAP output from standard output and not
-from standard error.  Lines written to standard output matching
address@hidden/^(not )?ok\b/} must be interpreted as test lines.  All other
-lines must not be considered test output.
address@hidden quotation
address@hidden @option
address@hidden --ignore-exit
+Causes the test driver to ignore the exit status of the test scripts;
+by default, the driver will report an error if the script exit with a
+non-zero status.
address@hidden --comments
+Instruct the test driver to display TAP diagnostic (i.e., lines beginning
+with the @samp{#} character) in the testsuite progress output too; by
+default, TAP diagnostic is only copied in the @file{.log} file.
address@hidden --no-comments
+Revert the effects of @option{--comments}.
address@hidden --merge
+Instruct the test driver to merge the test scripts' standard error into
+their standard output.  This is necessary if you want to ensure that
+diagnostics from the test scripts are displayed in the correct order
+relative to test results; this can be of great help in debugging
+(especially if your test scripts are shell scripts run with shell
+tracing active).  As a downside, this option might cause the test
+harness to get confused if anything that appears on standard error
+looks like a test result.
address@hidden --no-merge
+Revert the effects of @option{--merge}.
address@hidden table
 
 @noindent
-Here are some links to more extensive official or third-party documentation
-and resources:
+Here is an example of how the TAP driver can be set up and used.
+
address@hidden Keep in sync with tap-doc2.test.
address@hidden
+% @kbd{cat configure.ac}
+AC_INIT([GNU Try Tap], [1.0], [bug-automake@@gnu.org])
+AC_CONFIG_AUX_DIR([build-aux])
+AM_INIT_AUTOMAKE([foreign parallel-tests -Wall -Werror])
+AC_CONFIG_FILES([Makefile])
+AC_REQUIRE_AUX_FILE([tap-driver])
+AC_PATH_PROG([PERL], [perl])
+test -n "$PERL" || AC_MSG_ERROR([perl not found])
+$PERL -MTAP::Parser -e 1 || AC_MSG_ERROR([TAP::Parser not found])
+AC_OUTPUT
+
+% @kbd{cat Makefile.am}
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/build-aux/tap-driver
+TESTS = foo.test bar.test baz.test
+EXTRA_DIST = $(TESTS)
+
+% @kbd{cat foo.test}
+#!/bin/sh
+echo 1..4 # Number of tests to be executed.
+echo 'ok 1 - Swallows fly'
+echo 'not ok 2 - Caterpillars fly # TODO metamorphosis in progress'
+echo 'ok 3 - Pigs fly # SKIP not enough acid'
+echo '# I just love word plays ...'
+echo 'ok 4 - Flies fly too :-)'
+
+% @kbd{cat bar.test}
+#!/bin/sh
+echo 1..3
+echo 'not ok 1 - Bummer, this test has failed.'
+echo 'ok 2 - This passed though.'
+echo 'Bail out! Ennui kicking in, sorry...'
+echo 'ok 3 - This will not be seen.'
+
+% @kbd{cat baz.test}
+#!/bin/sh
+echo 1..1
+echo ok 1
+# Exit with error, even if all the test case has been successful.
+exit 7
+
+% @kbd{cp @var{PREFIX}/share/address@hidden/tap-driver .}
+% @kbd{autoreconf -vi && ./configure && make check}
+...
+PASS: foo.test 1 - Swallows fly
+XFAIL: foo.test 2 - Caterpillars fly # TODO metamorphosis in progress
+SKIP: foo.test 3 - Pigs fly # SKIP not enough acid
+PASS: foo.test 4 - Flies fly too :-)
+FAIL: bar.test 1 - Bummer, this test has failed.
+PASS: bar.test 2 - This passed though.
+ERROR: bar.test - Bail out! Ennui kicking in, sorry...
+PASS: baz.test 1
+ERROR: baz.test - exited with status 7
+...
+Please report to bug-automake@@gnu.org
+...
+% @kbd{echo exit status: $?}
+exit status: 1
+
address@hidden Keep the "skewed" indentation below, it produces pretty PDF 
output.
+% @kbd{env TEST_LOG_DRIVER_FLAGS='--comments --ignore-exit' \
+      TESTS='foo.test baz.test' make -e check}
+...
+PASS: foo.test 1 - Swallows fly
+XFAIL: foo.test 2 - Caterpillars fly # TODO metamorphosis in progress
+SKIP: foo.test 3 - Pigs fly # SKIP not enough acid
+# foo.test: I just love word plays...
+PASS: foo.test 4 - Flies fly too :-)
+PASS: baz.test 1
+...
+% @kbd{echo exit status: $?}
+exit status: 0
address@hidden example
+
address@hidden Incompatibilities with other TAP parsers and drivers
address@hidden Incompatibilities with other TAP parsers and drivers
+
+For implementation or historical reasons, the TAP driver and harness as
+implemented by Automake have some minors incompatibilities with the
+mainstream versions, which you should be aware of.
+
address@hidden @bullet
address@hidden
+A @code{Bail out!} directive doesn't stop the whole testsuite, but only
+the test script it occurs into.  This doesn't follows TAP specifications,
+but on the other hand it maximizes compatibility (and code sharing) with
+the ``hard error'' concept of the default @option{parallel-tests} driver.
address@hidden
address@hidden: there's surely something else ...
address@hidden itemize
+
address@hidden Links and external resources
address@hidden Links and external resources
+
address@hidden
+Here are some links to more extensive official or third-party
+documentation and resources:
 @itemize @bullet
 @item
 @uref{http://search.cpan.org/~petdance/Test-Harness/lib/Test/Harness/TAP.pod,
@@ -9565,39 +9740,9 @@ the standard perl testing libraries, which are based on 
TAP.
 @item
 @uref{http://www.eyrie.org/~eagle/software/c-tap-harness/,C TAP Harness},
 a C-based project implementing both a TAP producer and a TAP consumer.
address@hidden itemize
-
-Give example of output from @command{prove} command (when used in
-``verbose'' mode, with option @option{-v}), to which we'd like to
-remain somewhat compatible compatible, while also (and foremost)
-being compatible with the default @option{parallel-tests} driver.
-
-Example of output from out own TAP testsuite driver.  Point out
-similarities with the output from @command{prove --verbose} and
-from the default @option{parallel-tests} driver.
-
-Option @option{--ignore-exit} causes the driver to ignore the exit
-status of the test scripts; by default, the driver will report an
-error if the script exit with status != 0.
address@hidden: also add a @option{--no-ignore-exit} option, for
-completeness?
-
-By default, TAP diagnostic (i.e., lines beginning with the @samp{#}
-character) are copied only in the @file{.log} file.  The option
address@hidden causes the driver to display them in the testsuite
-progress output too (@emph{TODO}: give example).  The option
address@hidden restore the default behaviour.
-
address@hidden
-Differences and incompatibilities with other TAP parsers and drivers:
address@hidden @bullet
address@hidden
-A @code{Bail out!} directive doesn't stop the whole testsuite, but only
-the test script it occurs into.  This doesn't follows TAP specifications,
-but on the other hand maximize compatibility (and code sharing) with
-the ``hard error'' concept of the default @option{parallel-tests} driver.
 @item
address@hidden: there's surely something else ...
address@hidden://www.tap4j.org/,tap4j},
+a Java-based project implementing both a TAP producer and a TAP consumer.
 @end itemize
 
 @node DejaGnu Tests
diff --git a/tests/Makefile.am b/tests/Makefile.am
index b8126f2..6a45fcf 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -1184,6 +1184,7 @@ tap-bad-prog.test \
 tap-bad-prog2.test \
 tap-basic.test \
 tap-doc.test \
+tap-doc2.test \
 tap-empty.test \
 tap-more.test \
 tap-more2.test \
diff --git a/tests/Makefile.in b/tests/Makefile.in
index 82e62a3..a085762 100644
--- a/tests/Makefile.in
+++ b/tests/Makefile.in
@@ -1417,6 +1417,7 @@ tap-bad-prog.test \
 tap-bad-prog2.test \
 tap-basic.test \
 tap-doc.test \
+tap-doc2.test \
 tap-empty.test \
 tap-more.test \
 tap-more2.test \
diff --git a/tests/tap-doc2.test b/tests/tap-doc2.test
new file mode 100755
index 0000000..63bdc23
--- /dev/null
+++ b/tests/tap-doc2.test
@@ -0,0 +1,139 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Check that an example given in the documentation really works.
+# See section "Using the TAP test protocol", subsection "Use TAP
+# with the Automake test harness".
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+# Ensure we are run from the right directory.
+# (The last thing we want is to delete some random user files.)
+test -f ../defs-static && test -f ../defs || Exit 99
+rm -f *
+
+cat > Makefile.am <<'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/build-aux/tap-driver
+TESTS = foo.test bar.test baz.test
+EXTRA_DIST = $(TESTS)
+END
+
+cat > configure.ac <<'END'
+AC_INIT([GNU Try Tap], [1.0], address@hidden)
+AC_CONFIG_AUX_DIR([build-aux])
+AM_INIT_AUTOMAKE([foreign parallel-tests -Wall -Werror])
+AC_CONFIG_FILES([Makefile])
+AC_REQUIRE_AUX_FILE([tap-driver])
+AC_PATH_PROG([PERL], [perl])
+test -n "$PERL" || AC_MSG_ERROR([perl not found])
+$PERL -MTAP::Parser -e 1 || AC_MSG_ERROR([TAP::Parser not found])
+AC_OUTPUT
+END
+
+cat > foo.test <<'END'
+#!/bin/sh
+echo 1..4 # Number of tests to be executed.
+echo 'ok 1 - Swallows fly'
+echo 'not ok 2 - Caterpillars fly # TODO metamorphosis in progress'
+echo 'ok 3 - Pigs fly # SKIP not enough acid'
+echo '# I just love word plays...'
+echo 'ok 4 - Flies fly too :-)'
+END
+
+cat > bar.test <<'END'
+#!/bin/sh
+echo 1..3
+echo 'not ok 1 - Bummer, this test has failed.'
+echo 'ok 2 - This passed though.'
+echo 'Bail out! Ennui kicking in, sorry...'
+echo 'ok 3 - This will not be seen.'
+END
+
+cat > baz.test <<'END'
+#!/bin/sh
+echo 1..1
+echo ok 1
+# Exit with error, even if all the test case has been successful.
+exit 7
+END
+
+chmod a+x *.test
+
+mkdir build-aux
+cp "$top_testsrcdir"/lib/tap-driver build-aux
+
+(export AUTOMAKE ACLOCAL AUTOCONF && $AUTORECONF -vi) || Exit 1
+
+./configure --help # Sanity check.
+./configure || skip_ "configure failed"
+
+case $MAKE in *\ -j*) skip_ "can't work easily with concurrent make";; esac
+
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+cat > exp <<'END'
+PASS: foo.test 1 - Swallows fly
+XFAIL: foo.test 2 - Caterpillars fly # TODO metamorphosis in progress
+SKIP: foo.test 3 - Pigs fly # SKIP not enough acid
+PASS: foo.test 4 - Flies fly too :-)
+FAIL: bar.test 1 - Bummer, this test has failed.
+PASS: bar.test 2 - This passed though.
+ERROR: bar.test - Bail out! Ennui kicking in, sorry...
+PASS: baz.test 1
+ERROR: baz.test - exited with status 7
+END
+
+sed -n '/^PASS: foo\.test/,/^ERROR: baz\.test/p' stdout > got
+
+cat exp
+cat got
+diff exp got
+
+grep '^Please report to address@hidden' stdout
+
+env \
+  TESTS='foo.test baz.test' \
+  TEST_LOG_DRIVER_FLAGS='--comments --ignore-exit' \
+  $MAKE -e check >stdout || { cat stdout; Exit 1; }
+
+cat > exp <<'END'
+PASS: foo.test 1 - Swallows fly
+XFAIL: foo.test 2 - Caterpillars fly # TODO metamorphosis in progress
+SKIP: foo.test 3 - Pigs fly # SKIP not enough acid
+# foo.test: I just love word plays...
+PASS: foo.test 4 - Flies fly too :-)
+PASS: baz.test 1
+END
+
+sed -n '/^PASS: foo\.test/,/^PASS: baz\.test/p' stdout > got
+
+cat exp
+cat got
+diff exp got
+
+# Sanity check the distribution.
+cat > bar.test <<'END'
+#!/bin/sh
+echo 1..1
+echo ok 1
+END
+echo AM_TEST_LOG_DRIVER_FLAGS = --ignore-exit >> Makefile.in
+./config.status Makefile
+$MAKE distcheck
+
+:
-- 
1.7.2.3