Re: mauve results posted nightly

[Mark Wielaard]

Hi,

On Mon, 2002-11-11 at 07:04, Tom Tromey wrote:
> >>>>> "Brian" == Brian Jones <address@hidden> writes:
> 
> Brian> Please let me know if posting this information in this way is useful.
> Brian> Perhaps other JVMs could be added.  Does Gcj do this?
> 
> Mark Wielaard posts nightly test results to the gcc test mailing list.
> As I recall he does this for a couple platforms.  I usually just
> glance over these; a mailing list isn't that great of a format for
> detecting regressions.

I only post results of the stable (3.2.x) and development (3.3) branches
on powerpc (and sometimes for x86, if I feel like it, but that is not
automated).

The nice thing of posting to the gcc-testresults mailinglist is that you
have an archive of when what test passed/failed. This has helped me
figuring out when something broke and what patches to suspect.

If the Mauve Classpath+Kissme results could also be archived somehow
that would be really helpful for figuring out when what broke.

ftp://alpha.gnu.org/pub/gnu/classpath/nightly/tests/kissme-mauve-report.txtIt 
seems that standard out and standard error are somehow mixed up, which seems to 
put the ---- lines in the wrong places. You might want to use the recently 
introduced -resultsonly flag to only generate PASS or FAIL lines.

What would be really helpful is when our xfails and mauve-libgcj files
are completely correct and when we know for certain that the failing
tests in Mauve are real bugs in the implementation. Now the number of
false positives is a bit to high. My plan for after the gcj 3.3 code
freeze is to go over all the (x)failing (and commented out) Mauve tests
in libgcj and figure out if it is a real bug and file a bug report or to
fix the Mauve test. Last time we did that ,for gcj 3.1, we really
cleaned up a lot of bad mauve tests. Hopefully this time we can clean up
Mauve completely.

> Ben Elliston is working on a Tinderbox-like system.  It will include
> gcj test results.  Ideally it will also send out email when someone
> introduces a regression (like Geoff's old tester, but it will include
> libgcj).

That would be really great to have!

Cheers,

Mark