[Automake-commit] [SCM] GNU Automake branch, GSoC/experimental/test-results-work, created. v1.11-906-g118512c

[Stefano Lattarini]

This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "GNU Automake".

http://git.sv.gnu.org/gitweb/?p=automake.git;a=commitdiff;h=118512cadbfbf851a10a22e3611547feb99b5c8f

The branch, GSoC/experimental/test-results-work has been created
        at  118512cadbfbf851a10a22e3611547feb99b5c8f (commit)

- Log -----------------------------------------------------------------
commit 118512cadbfbf851a10a22e3611547feb99b5c8f
Author: Stefano Lattarini <address@hidden>
Date:   Sun Jul 24 16:47:24 2011 +0200

    simple tests: support developer-defined fd redirections
    
    Motivated by coreutils bug#8846, and related discussions:
     <http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8846>
     <http://lists.gnu.org/archive/html/bug-autoconf/2011-06/msg00002.html>
    In those threads it has been shown how problematic it can be to do
    portable file descriptor redirections in the testsuite when using
    the Automake testsuite harness.  This change should remedy to that
    situation.
    
    * lib/am/check2.am (?GENERIC?%EXT%$(EXEEXT).log,
    ?GENERIC?%EXT%.log, ?!GENERIC?%OBJ%): Append redirection defined
    in $(AM_TESTS_FD_REDIRECT) to the command-line invocations of the
    test scripts.
    * lib/am/check.am [!%?PARALLEL_TESTS%] $(check-TESTS): Likewise.
    * NEWS: Update.
    * tests/check-fd-redirect.test: New test.
    * tests/parallel-tests-fd-redirect.test: Likewise.
    * tests/Makefile.am (TESTS): Update.

commit c78b03137853bd6894e8e290c2604b8cd8bb3fd6
Author: Stefano Lattarini <address@hidden>
Date:   Thu Jul 21 17:07:54 2011 +0200

    docs: rework and extend documentation on testsuites support
    
    * doc/automake.texi (Generalities about Testing): New section.
    (Simple Tests using parallel-tests): Section removed, merged
    into ...
    (Simple Tests): ... this one, which on the other hand has been
    subdivided into ...
    (Script-based Testsuites, Serial Test Harness, Parallel Test
    Harness): ... these new subsection.
    (DejaGnu Tests): Minor adjustments.
    Other related typofixes and rewordings throughout the manual;
    in particular, avoid to use the term "test driver" for three
    different concepts (and use instead "test harness" and "test
    runner" where appropriate).
    * tests/tap-doc.test: New test.
    * tests/tap-no-disable-hard-error.test: Likewise.
    * tests/Makefile.am (TESTS): Update.

commit 415cd5bbc9ba07186db473540e0ac3d03dffb2fb
Author: Stefano Lattarini <address@hidden>
Date:   Sun Jul 17 16:46:41 2011 +0200

    tap: support colorization of testsuite progress output
    
    * lib/tap-driver (%COLORS): New variable (definition extracted
    from `lib/am/check.am:$(am__tty_colors)', with some obvious
    adjustments.
    (report): Adjust to colorize console output when required,
    using ...
    (decorate_result): ... this new function.
    (colored): New function, used by the one above.
    * tests/tap-summary.test: Also run the checks when `color-tests'
    is in use.
    * tests/Makefile.am (XFAIL_TESTS): Remove `tap-color.test'.

commit 38d225c86ab1dbcf28eecb3fb05501ca9e0430c7
Author: Stefano Lattarini <address@hidden>
Date:   Sun Jul 17 16:04:06 2011 +0200

    tap: some preparatory refactoring (2)
    
    This is a follow-up simplification.
    
    * lib/tap-driver (console_output): Renamed ...
    (report): ... to this, and extended to appropriately register
    the test results when needed.
    (testsuite_error, handle_tap_comment, handle_tap_test,
    handle_tap_plan): Adjusted accordingly.

commit 43b4a92e3dc8b2dafc9b828d590954aba379be5b
Author: Stefano Lattarini <address@hidden>
Date:   Sun Jul 17 15:18:51 2011 +0200

    tap: some preparatory refactoring (1)
    
    This refactoring is aimed at simplifying the introduction of
    colored console output for the TAP driver.
    
    * lib/tap-driver (console_output): Now accept two arguments, the
    first one indicating which kind of thing is to be displayed (for
    now only a test result or a diagnostic comment), and the second
    one (if present) the message associated to it.
    (handle_tap_test, handle_tap_comment, handle_tap_plan,
    testsuite_error): Adapt to the new `console_output' interface.

commit c761822446ef1421ccc57c902b00cb5f804f7dd9
Author: Stefano Lattarini <address@hidden>
Date:   Sun Jul 3 19:20:22 2011 +0200

    tap: add experimental TAP-aware driver
    
    * doc/automake.texi (Using the TAP test protocol): New section.
    (Overview of Custom Test Drivers Support): Minor updates.
    * lib/tap-driver: New script, TAP-aware test driver for Automake;
    implemented in perl and based on TAP::Parser.
    * lib/Makefile.am (dist_script_DATA): Add it.
    * tests/tap-autonumber.test: New test.
    * tests/tap-bailout.test: Likewise.
    * tests/tap-basic.test: Likewise.
    * tests/tap-deps.test: Likewise.
    * tests/tap-diagnostic.test: Likewise.
    * tests/tap-empty.test: Likewise.
    * tests/tap-empty-diagnostic.test: Likewise.
    * tests/tap-escape-directive.test: Likewise.
    * tests/tap-exit.test: Likewise.
    * tests/tap-fancy.test: Likewise.
    * tests/tap-fancy2.test: Likewise.
    * tests/tap-global-result.test: Likewise.
    * tests/tap-html.test: Likewise.
    * tests/tap-log.test: Likewise.
    * tests/tap-merge-stdout-stderr.test: Likewise.
    * tests/tap-more.test: Likewise.
    * tests/tap-more2.test: Likewise.
    * tests/tap-no-merge-stdout-stderr.test: Likewise.
    * tests/tap-no-spurious-summary.test: Likewise.
    * tests/tap-no-spurious.test: Likewise.
    * tests/tap-not-ok-skip.test: Likewise.
    * tests/tap-numeric-description.test: Likewise.
    * tests/tap-out-of-order.test: Likewise.
    * tests/tap-passthrough.test: Likewise.
    * tests/tap-plan.test: Likewise.
    * tests/tap-plan-errors.test: Likewise.
    * tests/tap-plan-corner.test: Likewise.
    * tests/tap-realtime.test: Likewise.
    * tests/tap-recheck-logs.test: Likewise.
    * tests/tap-recheck.test: Likewise.
    * tests/tap-skip-whole.test: Likewise.
    * tests/tap-summary.test: Likewise.
    * tests/tap-todo-skip.test: Likewise.
    * tests/tap-todo-skip-together.test: Likewise.
    * tests/tap-todo-skip-whitespace.test: Likewise.
    * tests/tap-skipall-whitespace.test: Likewise.
    * tests/tap-unplanned.test: Likewise.
    * tests/tap-whitespace-normalization.test: Likewise.
    * tests/tap-with-and-without-number.test: Likewise.
    * tests/tap-xfail-tests.test: Likewise.
    * tests/tap-bad-prog.test: New xfailing test.
    * tests/tap-color.test: Likewise.
    * tests/tap-plan-corner2.test: Likewise.
    * tests/tap-message-0.test: Likewise.
    * tests/tap-signal.test: Likewise.
    * tests/Makefile.am (TESTS, XFAIL_TESTS): Update.

commit 505746496d1ff30d423d3759cdfd1792d1be3420
Author: Stefano Lattarini <address@hidden>
Date:   Mon Jul 4 18:12:07 2011 +0200

    tests defs: new auxiliary function 'count_test_results'
    
    * tests/defs (count_test_results): New function.
    * tests/check11.test: Use it.
    * tests/test-driver-custom-multitest.test: Likewise.
    * tests/test-driver-custom-multitest-recheck.test: Likewise.
    * tests/test-driver-custom-multitest-recheck2.test: Likewise.
    * tests/parallel-tests-log-override-recheck.test: Likewise.
    * tests/parallel-tests-log-override-recheck.test: Likewise.
    * tests/parallel-tests-no-spurious-summary.test: Likewise, and
    slightly improve debugging output.
    * tests/parallel-tests.test: Make use of `count_test_results'.
    Also, make grepping of "make check" output slightly stricter
    * tests/parallel-tests9.test: Likewise.
    * tests/parallel-tests-log-override-2.test: Likewise, and throw
    in a small optimization.

commit eaac33fe13811cf104f45a09a6ece911c135fa83
Author: Stefano Lattarini <address@hidden>
Date:   Fri Jul 1 11:12:31 2011 +0200

    parallel-tests: simplify testsuite summary
    
    Prefer a more deterministic, "tabular" format for the testsuite
    summary, always listing the numbers of passed, failed, xfailed,
    xpassed, skipped and errored tests, even when these numbers are
    zero.  This simplify the logic of testsuite summary creation,
    makes it more easily machine-parseable, and will probably allow
    for easier addition of new kinds of test results in the future.
    
    * lib/am/check.am (am__tty_colors_dummy): New make variable, to
    reduce code duplication.  Extracted from previous versions of
    $(am__tty_colors), and extended by defining two new variables
    `$mgn' and `$brg'.
    [%?COLOR%, %!?COLOR%] (am__tty_colors): Use that new variable.
    (am__text_box): Delete, is not needed anymore.
    ($(TEST_SUITE_LOG)): Rewrite associated rules to implement the
    new testsuite summary format.
    * NEWS: Update.
    * tests/check10.test: Don't run with the parallel-tests harness
    too, that makes no sense anymore.
    * tests/color.test: Update and adjust.
    * tests/color2.test: Likewise.
    * tests/parallel-tests.test: Likewise.
    * tests/parallel-tests3.test: Likewise.
    * tests/parallel-tests6.test: Likewise.
    * tests/parallel-tests9.test: Likewise.
    * tests/parallel-tests-unreadable-log.test: Likewise.
    * tests/parallel-tests-empty-testlogs.test: Likewise.
    * tests/parallel-tests-log-override-recheck.test: Likewise.
    * tests/parallel-tests-no-spurious-summary.test: Likewise.
    * tests/test-driver-custom-multitest.test: Likewise.
    * tests/test-driver-end-test-results.test: Likewise.
    * tests/parallel-tests-no-color-in-log.test: New test.
    * tests/testsuite-summary-color.test: Likewise.
    * tests/testsuite-summary-count.test: Likewise.
    * tests/testsuite-summary-count-many.test: Likewise.
    * tests/testsuite-summary-reference-log.test: Likewise.
    * tests/testsuite-summary-checks.sh: New auxiliary script, used
    by the new tests above.
    * tests/extract-testsuite-summary: Likewise.
    * tests/trivial-test-driver: Optimize for speed when there are
    lots of of tests.
    * tests/Makefile.am (EXTRA_DIST): Distribute them.
    (testsuite-summary-color.log, testsuite-summary-count.log): Depend
    on them.
    (testsuite-summary-count-many.log): Depend on the auxiliary scripts
    'trivial-test-driver' and 'extract-testsuite-summary'.
    (TESTS): Update.

commit 15400bca303910d059f1a68f94b25cffb1a8ac2f
Author: Stefano Lattarini <address@hidden>
Date:   Thu Jun 30 14:55:43 2011 +0200

    parallel-tests: new recognized test result 'ERROR'
    
    * lib/am/check.am ($(TEST_SUITE_LOG)): Recognize a new test result
    `ERROR'.  Use it when encountering unreadable test logs (previously
    a simple `FAIL' was used in this situations).
    * lib/test-driver: Set the global test result to `ERROR' when the
    test exit status is 99.  When doing colorized output, color `ERROR'
    results in magenta.
    * doc/automake.texi (Log files generation and test results
    recording): Update by also listing `ERROR' among the list of valid
    `:test-results:' arguments.
    * NEWS: Update.
    * tests/trivial-test-driver: Update.
    * tests/parallel-tests.test: Likewise.
    * tests/parallel-tests-harderror.test: Likewise.
    * tests/parallel-tests-no-spurious-summary.test: Likewise.
    * tests/test-driver-global-log.test: Likewise.
    * tests/test-driver-recheck.test: Likewise.
    * tests/test-driver-custom-multitest-recheck.test: Likewise.
    * tests/test-driver-custom-multitest-recheck2.test: Likewise.
    * tests/test-driver-custom-multitest.test: Likewise.
    * tests/test-driver-custom-no-html.test: Likewise.
    * tests/test-driver-end-test-results.test: Likewise.
    * tests/color.test: Likewise.  Also, make stricter, and also test
    from VPATH.
    * tests/color2.test: Likewise, and improve syncing with color.test.
    * tests/parallel-tests-exit-statuses.test: New test.
    * tests/parallel-tests-console-output.test: Likewise.
    * tests/Makefile.am (TESTS): Update.

commit 61b2bb0d647ea5f192e49f5f5885009ea83968be
Author: Stefano Lattarini <address@hidden>
Date:   Wed Jun 29 16:02:52 2011 +0200

    parallel-tests: make parsing of test results safer
    
    The new code for parsing the testsuite-generated `.log' files,
    as introduced in commit `v1.11-872-gc96b881', considers each
    `:test-result:' field anywhere in a `.log' file as a declaration
    of a test result, and accounts for it as such in the testsuite
    summary.  Unfortunately this could easily cause spurious test
    failures being reported in the testsuite summary.  This happened
    in practice with the Automake's own testsuite; for example:
    
      $ make check TESTS='check12-p.test'; echo exit: $?
      ...
      PASS: check12-p.test
      =====================================
      4 of 5 tests failed
      See tests/test-suite.log
      Please report to address@hidden
      =====================================
      make[2]: *** [test-suite.log] Error 1
      make: *** [check-am] Error 2
      exit: 2
    
    This change introduces a new special `:test-result:' "END", that,
    when seen, prevents the rest of the log file from being parsed.
    
    For more information, refer to the thread:
    <http://lists.gnu.org/archive/html/automake-patches/2011-06/msg00199.html>
    
    * lib/am/check.am ($(TEST_SUITE_LOG)): Stop the parsing of a log
    file as soon as the special ":test-result:END" directive is seen.
    Related changes and enhancements.
    * lib/test-driver: Protect the rest of the log after the result
    lined with a ":test-result:END" directive.
    * tests/parallel-tests-no-spurious-summary.test: New test.
    * tests/test-driver-end-test-results.test: Likewise.
    * tests/Makefile.am (TESTS): Update.

-----------------------------------------------------------------------


hooks/post-receive
-- 
GNU Automake