Re: Mauve test question

[Archie Cobbs]

Jeroen Frijters wrote:
>>Huh? Why is adding broken tests the right thing to do?
> 
> The test isn't broken, it just exposes a bug (or missing feature) in
> Classpath. Often when you run into a Classpath bug (or missing feature)
> the right thing to do is to add a test, later Classpath gets fixed.
> 
>>And besides, if a broken test is added, this way there will be
>>motivation to resolve the discrepancy. With a whitelist, a broken
>>test can get added but no one will notice and then it just sits
>>there getting stale.
> 
> The alternative is worse, it's the situation we're in now. There are
> lots of broken tests and we have no idea if the failures are due to
> Classpath, VM or test bugs.

I agree the situation now is bad, that is what I'm trying to fix.

The problem you describe with a blacklist goes away when there is
also an "xfails" file. I think the reason you want a whitelist is
because you use ./batch_run, which doesn't support "xfails". This
appears to me to be a deficiency of ./batch_run, not proof that a
whitelist is better.

Consider this process:

- At any given time, "mauve-classpath" contains instructions to run
   all tests except those that are known completely broken (eg unicode),
   and Classpath's "xfails" contains known Classpath test failures among
   the tests specified in "mauve-classpath".

- In the steady state, running mauve with "mauve-classpath" and Classpath's
   "xfails" produces ZERO unexpected failures, except those that are specific
   to your JVM (you may have your own private "xfails" to account for this).
   Any failures it does produce are obvious and meaningful.

- If a new test is added to Mauve, it automatically gets inclued in the
   Classpath test suite. If Classpath fails the test due to a Classpath bug,
   then the failure is immediately obvious, and someone then will either fix
   the bug or add the test to "xfails" (effectively putting it on the "to-do"
   list for Classpath hackers). In either case, because the new test failure
   stands out, it will get categorized immediately.

- If a bug fix in Classpath fixes an "xfails" test, that will appear as
   an XPASS "unexpected success" and can then be removed from xfails.
   Immediately it becomes a regression test.

In this way we maintain our test status continuously. All tests are
categorized as either "don't run", "should pass", or "expect failure".

> There is obviously no silver bullet, it a matter of making the right
> trade-off. I believe I white list is the right trade-off, because it
> offers a list that is known to be good at all times, so when someone
> checks out Classpath and starts hacking, she will be comfortable to know
> that the white list will contain tests that run correctly and can use
> that fact to test if Classpath and/or the VM were build correctly and
> any changes made didn't break anything (subject to test coverage, of
> course).

This is also exactly what "mauve-classpath" plus Classpath's "xfails"
gives you, but a whitelist is harder to maintain.

> The downside of a white list is that we need to be extra vigilant in
> keeping it up to date, but I don't think this is a problem. It might be
> helpful to have an option in Mauve to list all the successful tests in a
> particular run, that way that list can be compared against the standard
> white list to see if additional tests can be added to the white list.

Yes, that is the downside :-)

-Archie

__________________________________________________________________________
Archie Cobbs      *        CTO, Awarix        *      http://www.awarix.com


*
Confidentiality Notice: This e-mail message, including any attachments, is for 
the sole use of the intended recipient(s) and may contain confidential and 
privileged information. Any unauthorized review, use, disclosure or 
distribution is prohibited. If you are not the intended
recipient, please contact the sender by reply e-mail and destroy all copies of 
the original message.
*