RE: Mauve test question

[Jeroen Frijters]

Archie Cobbs wrote:
> Jeroen Frijters wrote:
> >>The problem with that approach is that if someone adds a new test
> >>to Mauve, it doesn't automatically get added to our "white list".
> > 
> > That's a feature, not a bug! In practice new tests often 
> get added that
> > don't yet run without failures (and this is the right thing 
> to do). So I
> > strongly believe we should work with a white list.
> 
> Huh? Why is adding broken tests the right thing to do?

The test isn't broken, it just exposes a bug (or missing feature) in
Classpath. Often when you run into a Classpath bug (or missing feature)
the right thing to do is to add a test, later Classpath gets fixed.

> And besides, if a broken test is added, this way there will be
> motivation to resolve the discrepancy. With a whitelist, a broken
> test can get added but no one will notice and then it just sits
> there getting stale.

The alternative is worse, it's the situation we're in now. There are
lots of broken tests and we have no idea if the failures are due to
Classpath, VM or test bugs.

There is obviously no silver bullet, it a matter of making the right
trade-off. I believe I white list is the right trade-off, because it
offers a list that is known to be good at all times, so when someone
checks out Classpath and starts hacking, she will be comfortable to know
that the white list will contain tests that run correctly and can use
that fact to test if Classpath and/or the VM were build correctly and
any changes made didn't break anything (subject to test coverage, of
course).

The downside of a white list is that we need to be extra vigilant in
keeping it up to date, but I don't think this is a problem. It might be
helpful to have an option in Mauve to list all the successful tests in a
particular run, that way that list can be compared against the standard
white list to see if additional tests can be added to the white list.

Regards,
Jeroen