[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [PATCH 2/5] {test-protocols} parallel-tests: new recognized test res
From: |
Stefano Lattarini |
Subject: |
Re: [PATCH 2/5] {test-protocols} parallel-tests: new recognized test result 'ERROR' |
Date: |
Tue, 19 Jul 2011 11:38:18 +0200 |
User-agent: |
KMail/1.13.3 (Linux/2.6.30-2-686; KDE/4.4.4; i686; ; ) |
On Monday 18 July 2011, Ralf Wildenhues wrote:
> * Stefano Lattarini wrote on Fri, Jul 15, 2011 at 12:37:07AM CEST:
> > * lib/am/check.am ($(TEST_SUITE_LOG)): Recognize a new test result
> > `ERROR'. Use it when encountering unreadable test logs (previously
> > a simple `FAIL' was used in this situations).
> > * lib/test-driver: Set the global test result to `ERROR' when the
> > test exit status is 99. When doing colorized output, color `ERROR'
> > results in magenta.
> > * doc/automake.texi (Log files generation and test results
> > recording): Update by also listing `ERROR' among the list of valid
> > `:test-results:' arguments.
> > * NEWS: Update.
> > * tests/trivial-test-driver: Update.
> > * tests/parallel-tests.test: Likewise.
> > * tests/parallel-tests-harderror.test: Likewise.
> > * tests/parallel-tests-no-spurious-summary.test: Likewise.
> > * tests/test-driver-global-log.test: Likewise.
> > * tests/test-driver-recheck.test: Likewise.
> > * tests/test-driver-custom-multitest-recheck.test: Likewise.
> > * tests/test-driver-custom-multitest-recheck2.test: Likewise.
> > * tests/test-driver-custom-multitest.test: Likewise.
> > * tests/test-driver-custom-no-html.test: Likewise.
> > * tests/test-driver-end-test-results.test: Likewise.
> > * tests/color.test: Likewise. Also, make stricter, and also test
> > from VPATH.
> > * tests/color2.test: Likewise, and improve syncing with color.test.
> > * tests/parallel-tests-exit-statuses.test: New test.
> > * tests/parallel-tests-console-output.test: Likewise.
> > * tests/Makefile.am (TESTS): Update.
>
> > --- a/NEWS
> > +++ b/NEWS
> > @@ -13,6 +13,10 @@ New in 1.11a:
> >
> > * Changes to Automake-generated testsuite harnesses:
> >
> > + - The parallel-tests harness has a new test result 'ERROR', that can be
> > + used to signal e.g., unexpected or internal errors, or failures to set
> > + up test case scenarios.
>
> The NEWS entry could be a bit more precise in that the ERROR state was
> actually used before already with exit status 99, just that it is named
> ERROR now.
>
What about this?
- Test scripts that exit with status 99 to signal an "hard error" (e.g.,
and unexpected or internal error, or a failure to set up the test case
scenario) have their outcome reported as an 'ERROR' now. Previous
versions of automake reported such an outcome as a 'FAIL' (the only
difference with normal failures being that hard errors were counted
as failures even when the test originating them was listed in
XFAIL_TESTS).
> (Please check whether the semantics of 99 already were in a stable release,
>
Well, an exit status of `99' has been triggering an hard error since at
least automake 1.11 (assuming `parallel-tests' was in use). Is this what
you were asking, or have I misunderstood?
> otherwise this NEWS entry could of course advertise it
> as new.)
>
> A description of ERROR semantics needs to be part of automake.texi as
> well.
>
Doing that properly will require a small reorganization of the "Support
for test suites" chapter, so I'd rather do that in a follow-up patch
(which I will start working on shortly).
> Also, are you going to followup with Autoconf to rename Autotest's hard
> failure state ERROR as well? We should agree on common naming and
> semantics, so the two systems are not harder to learn than necessary.
>
Good idea, consistency is always nice.
> I'm OK with this change once these issues are resolved.
>
> As a minor detail however, please remove the '====' greps, I've already
> mentioned why I consider '=' in the output not a good idea.
>
But those greps are there to ensure that the testsuite summary is not
unduly colorized; and since the summary at this point is still looks
like:
====================================================
All 2 tests behaved as expected (1 expected failure)
====================================================
those grepping checks have to stay IMNSHO.
> Thanks,
> Ralf
>
Regards,
Stefano
- Re: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer, (continued)
- Re: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer, Ralf Wildenhues, 2011/07/18
- Re: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer, Stefano Lattarini, 2011/07/19
- Re: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer, Ralf Wildenhues, 2011/07/19
- Re: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer, Stefano Lattarini, 2011/07/20
- Re: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer, Ralf Wildenhues, 2011/07/20
- Re: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer, Stefano Lattarini, 2011/07/21
- [PATCH 1/2] test harness: allow more metadata in log files, Stefano Lattarini, 2011/07/29
- Re: [PATCH 1/2] test harness: allow more metadata in log files, Stefano Lattarini, 2011/07/31
[PATCH 2/5] {test-protocols} parallel-tests: new recognized test result 'ERROR', Stefano Lattarini, 2011/07/14
[PATCH 3/5] {test-protocols} parallel-tests: simplify testsuite summary, Stefano Lattarini, 2011/07/14
[PATCH 4/5] {test-protocols} tests defs: new auxiliary function 'count_test_results', Stefano Lattarini, 2011/07/14
[PATCH 5/5] {test-protocols} tap: add experimental TAP-aware driver, Stefano Lattarini, 2011/07/14
Re: [GSoC] Some patches for testsuite harness improvements and TAP support introduction, Ralf Wildenhues, 2011/07/18