Re: mauve results posted nightly

[Stephen Crawley]

Hi Brian,

> So barring accidents of nature or human error I'll hopefully be able
> to post nightly Mauve results for Kissme/Classpath starting today.
> The Classpath homepage, http://www.classpath.org/, has been updated
> with links.  I have added japicompat results against 1.1 and updated
> to japitools-0.9.
> 
> Please let me know if posting this information in this way is useful.

I think that this is VERY useful.

> Perhaps other JVMs could be added.  Does Gcj do this?

Maybe a Sun JDK 1.4.x for reference?

If you have time to work on this, it would be a good idea if the web page 
and results defined the testing methodology, to aid people in reproducing
the test results.

* The log report should include datestamps for the latest sync with
Classpath, Kissme and Mauve CVS repositries.  

* the web page should document the hardware and OS platform/version, 
the C & Java compilers and versions, and versions of key libraries used 
(e.g. glibc).

* The Mauve "keys" file used should be linked to the page, or available
some other way.  It should be version controlled, as it will surely change
over time.

Here are some other suggestions:

*  The web page could explain that there are many possible causes of test
"failures"; e.g. Classpath bugs or incomplete feautures, Kissme bugs,
Mauve test case bugs, and (even) Sun JDK bugs and incompatibilities.
[Also, note the possibility of network failures, and failures caused by
non-atomic CVS commits.]

*  The nightly test results should be archived, with the archive ideally
accessible via the web page.  It would be very nice if the web page
could graph the numbers of tests passing and failing over time!! But
that could be a lot of work to set up.

-- Steve