Re: Mauve test question

[Thomas Zander]

On Tuesday 28 December 2004 17:54, Jeroen Frijters wrote:
> > And besides, if a broken test is added, this way there will be
> > motivation to resolve the discrepancy. With a whitelist, a broken
> > test can get added but no one will notice and then it just sits
> > there getting stale.
>
> The alternative is worse, it's the situation we're in now. There are
> lots of broken tests and we have no idea if the failures are due to
> Classpath, VM or test bugs.

I may be mistaken, and all this is way more complicated then I think. But 
please bear with me;

Just about all Mauve tests run correctly on Suns JVM, with a few exceptions 
since that one has bugs to.  Agree?
What about making that the 'baseline' test, with a file noting the specific 
exceptions.   Something like;
gnu.testlet.java.awt.RenderingHints.containsKey:4 = 1.0 - 1.4.2_05
meaning that test is expected to fail on the baseline JVM (Suns) from 
version 1.0 up-to-1.4.2_05

With that in place; all failing tests are bugs in the JVMs and/or Classpath.

Next;
lets write a test-runner that automatically creates one jar file (I just 
finished doing so, its 1Mb) with all tests from java.* and javax.* and then 
places that on a server, with the test-run-results from a couple of Sun 
JVMs.
I could probably put that into a runnable jar where you can pass arguments 
like version and other options taken from the source files.

Other JVMs either download that jar, or do a nightly/weekly compile on the 
same server and also publish their results in a similar file.

Next; write an application that shows which tests fail on which JVM using 
all those test-results and draw conclusions.  If a test fails on _all_ 
JVMs, except Suns, its probably a bug in Classpath.

Did I misunderstand the conversation here?  Or is this a much easier way to 
go?
I'm willing to put up a jar somewhere (can I scp it to a red-hat server?)  
since I seem to be one of the few who has Suns JVM(s) and ant installed :)

jar:  http://members.home.nl/zander/alltests.jar (1mb)
-- 
Thomas Zander

pgpSJ_0SHaJ2f.pgp
Description: PGP signature