[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [PATCH 2/5] {test-protocols} parallel-tests: new recognized test res
From: |
Ralf Wildenhues |
Subject: |
Re: [PATCH 2/5] {test-protocols} parallel-tests: new recognized test result 'ERROR' |
Date: |
Tue, 19 Jul 2011 21:16:35 +0200 |
* Stefano Lattarini wrote on Tue, Jul 19, 2011 at 11:38:18AM CEST:
> On Monday 18 July 2011, Ralf Wildenhues wrote:
> > * Stefano Lattarini wrote on Fri, Jul 15, 2011 at 12:37:07AM CEST:
> > > + - The parallel-tests harness has a new test result 'ERROR', that can be
> > > + used to signal e.g., unexpected or internal errors, or failures to
> > > set
> > > + up test case scenarios.
> >
> > The NEWS entry could be a bit more precise in that the ERROR state was
> > actually used before already with exit status 99, just that it is named
> > ERROR now.
> >
> What about this?
>
> - Test scripts that exit with status 99 to signal an "hard error" (e.g.,
> and unexpected or internal error, or a failure to set up the test case
> scenario) have their outcome reported as an 'ERROR' now. Previous
> versions of automake reported such an outcome as a 'FAIL' (the only
> difference with normal failures being that hard errors were counted
> as failures even when the test originating them was listed in
> XFAIL_TESTS).
Cool.
> > (Please check whether the semantics of 99 already were in a stable release,
> >
> Well, an exit status of `99' has been triggering an hard error since at
> least automake 1.11 (assuming `parallel-tests' was in use). Is this what
> you were asking,
Yup, thanks.
> or have I misunderstood?
> > A description of ERROR semantics needs to be part of automake.texi as
> > well.
> >
> Doing that properly will require a small reorganization of the "Support
> for test suites" chapter, so I'd rather do that in a follow-up patch
> (which I will start working on shortly).
OK. Just so we don't forget.
> > Also, are you going to followup with Autoconf to rename Autotest's hard
> > failure state ERROR as well? We should agree on common naming and
> > semantics, so the two systems are not harder to learn than necessary.
> >
> Good idea, consistency is always nice.
Thanks.
> > I'm OK with this change once these issues are resolved.
> >
> > As a minor detail however, please remove the '====' greps, I've already
> > mentioned why I consider '=' in the output not a good idea.
> >
> But those greps are there to ensure that the testsuite summary is not
> unduly colorized; and since the summary at this point is still looks
> like:
>
> ====================================================
> All 2 tests behaved as expected (1 expected failure)
> ====================================================
>
> those grepping checks have to stay IMNSHO.
Okeydokey for now. Except of course that I suggested to not have '='
signs in the new output format. ;-)
Cheers,
Ralf
- Re: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer, (continued)
- Re: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer, Stefano Lattarini, 2011/07/19
- Re: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer, Ralf Wildenhues, 2011/07/19
- Re: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer, Stefano Lattarini, 2011/07/20
- Re: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer, Ralf Wildenhues, 2011/07/20
- Re: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer, Stefano Lattarini, 2011/07/21
- [PATCH 1/2] test harness: allow more metadata in log files, Stefano Lattarini, 2011/07/29
- Re: [PATCH 1/2] test harness: allow more metadata in log files, Stefano Lattarini, 2011/07/31
[PATCH 2/5] {test-protocols} parallel-tests: new recognized test result 'ERROR', Stefano Lattarini, 2011/07/14
[PATCH 3/5] {test-protocols} parallel-tests: simplify testsuite summary, Stefano Lattarini, 2011/07/14
[PATCH 4/5] {test-protocols} tests defs: new auxiliary function 'count_test_results', Stefano Lattarini, 2011/07/14
[PATCH 5/5] {test-protocols} tap: add experimental TAP-aware driver, Stefano Lattarini, 2011/07/14
Re: [GSoC] Some patches for testsuite harness improvements and TAP support introduction, Ralf Wildenhues, 2011/07/18