continuous testing

[Tom Tromey]

Mark has set up a new classpath-testresults list, listed here:

http://savannah.gnu.org/mail/?group=classpath

The intent is for automated test results to be sent here, so that we
can track the state of the code more closely.  I would like to suggest
that developers monitor the list and fix regressions that they
introduce.


Currently the only tester sending results there is the one I'm running
here at Red Hat.  It tests a number of things continuously:

* Builds and checks gcj cvs head (including mauve and jacks)
* Builds gcjx, runs jacks against it
* Builds classpath with both the just-built gcj and gcjx
* Builds jamvm
* Builds mauve (using gcjx), runs against the just-built jamvm and
  classpath
* Builds classpath generics branch with ecj

It is configured to send email on a build failure, or if there is a
regression.  The baseline for regressions is set by the first run of
the test suite, and if a run completes with no regressions a new
baseline is set (this matters if more tests are added -- at the
moment, a new failing test is not considered a 'regression').  (Note
that it is trivial to reset the baseline if we run into a situation
where a regression is deemed acceptable.)

There is also a little ircbot, named cpbot, which can serve these
results on #classpath and #gcj.  Try '=help' for info.

I'm hoping to check the test scripts (and bot source) into a new
module in the classpath cvs repository sometime soon.

If you have suggestions for new things to test, or for features to add
to the bot, I'm all ears.  I can't promise to implement everything,
but that's why I'm going to put it into cvs :-).  I'm also hoping to
move the actual testing to a public machine so that other people can
change the running code.

A few existing ideas for improvements:

* build and run gjdoc
* run application test suites, eg kawa
* run mauve serialization test suite
* set up a japi run and write a script to generate a report of
  japi regressions

Tom