Re: Mauve test question

[Archie Cobbs]

Thomas Zander wrote:
>>>Huh? Why is adding broken tests the right thing to do? And besides,
>>>if a broken test is added, this way there will be motivation to
>>>resolve the discrepancy. With a whitelist, a broken test can get
>>>added but no one will notice and then it just sits there getting
>>>stale.
>>
>>Its common practise to add new code to one implementation, e.g GNU
>>classpath or libgcj, and test it for a while and later merge it to
>>kaffe. According to you the mauve tests don't need to be added before
>>it's included in all implementations because nothing may be broken.
> 
> 
> Ehm; just being a bystander;  a broken test in Archies email is a test that 
> does not work properly (harness.check(1 ==2)).
> A broken test we are talking about, and what Michael seems to imply; is a 
> test that is fully correct, and will (probably) run correctly on Suns JVM, 
> but fails on another.
> 
> Lets call the former a broken, and the latter a failing test, please :)
> 
> Mauve is not suppost to hold _any_ broken tests, right?

Thanks, I was misreading the original comment. We all agree Mauve
may contain "failing" tests but should never contain "broken" tests.

Now, back to the original point... I've made a proposal for cleaning up the 
Mauve mess. For those who don't like it, please explain your proposal and 
most importantly exactly how this "whitelist" (or whatver) will be 
maintained. Who is going to do the maintenance work? When are they going to 
do it? Etc.

I don't really care how we do it, but these seem to be reasonable
requirements. Maybe we should try to agree on these first:

- It should be possible to test any JVM using some "official" set of
   tests which a Classpath JVM should pass and see clearly and
   obviously any failures; a "perfect" JVM would have no failures.

- When a new test is added to Mauve, by default it is automatically
   added to the set of Classpath tests. I.e., it's not possible for
   newly added tests to not actually run against Classpath without
   explicitly configuring things that way.

The point is that every test has a known state, one of:

(a) don't even bother trying it (doesn't compile or we know it fails)
(b) it is a known failure (i.e., a Classpath bug)
(c) it should pass (i.e., if it fails, it must be a JVM bug)
(d) it should pass but there can be false negatives depending on the JVM

and most importantly the maintenance of these states is simle enough
that we don't lose track and get back into the mess we're in now.

By the way, saying "fix ./batch_run to respect xfails" or "fix the
normal mauve run to not ignore tests that don't compile" is fair
game (as long as you are willing to do the work :-)

-Archie

__________________________________________________________________________
Archie Cobbs      *        CTO, Awarix        *      http://www.awarix.com


*
Confidentiality Notice: This e-mail message, including any attachments, is for 
the sole use of the intended recipient(s) and may contain confidential and 
privileged information. Any unauthorized review, use, disclosure or 
distribution is prohibited. If you are not the intended
recipient, please contact the sender by reply e-mail and destroy all copies of 
the original message.
*