Re: Parallel tests execution [0/4]

[Akim Demaille]

Le 18 oct. 08 à 07:37, Ralf Wildenhues a écrit :

> * Akim Demaille wrote on Fri, Oct 17, 2008 at 11:12:45AM CEST:
> > At some point we were developping more-or-less test driven: many  
> > tests
> > were written way before it was possible to pass them, so we knew that
> > the tests would fail and they were XFAIL'ed.  Yet, it was not  
> > acceptable
> > for the executable being tested to die with a SEGV.  So simple  
> > failures
> > are just exit 1, and hard-errors (or 'unacceptable errors') had an  
> > exit
> > status that could not be saved by XFAIL.
>
> Hmm.  Ah, ok.
>
> So the desired semantics are: we have a test that is marked as XFAIL
> currently, but if it does this and that, then that should be a FAIL
> because it is worse than the expected failure.

Correct.

> Yes, that sounds interesting.  And it is different from my proposed
> semantics:
>
> We have a failure that should prevent further test from even being
> tried.

Yes, I have understood that.  And it sounds like an interesting feature.


> Now, the implementation of my proposed semantics /almost/ allows to  
> have
> your proposed semantics, too: you can run
>  make -k check

That's really different: you now require the user, who runs the test  
suite, to change the way he should run it.


> and the driver will run all tests, even after hard failures.  In the
> presence of hard failures, however, it will fail to produce a summary
> and the test-suite.log file.

Exactly.  This is very wrong.

> I'm not sure whether I want to introduce both semantics.  I see  
> yours as
> reasonable, I'm not sure how applicable mine would be in practice.
> Opinions appreciated.

Your semantics can be addressed via dependencies, don't it?

In my test suites I do have the case of tests that cannot succeed  
under some conditions, but n that case they all individually exit  
SKIP.  That way, even if you run just some of the tests (via explicit  
make check TESTS=...), they will be ignored, even if you skipped some  
of the HARD error (in your sense) tests.