Re: benchmarks (was Re: Progress on a Classpath mauve suite?)

[Chris Pickett]

David P Grove wrote:
> I think you need to understand the point of the benchmark suite.  The 
> whole goal is reproducible science, so if someone doesn't cite the exact 
> version of the benchmarks, then it isn't useful (in an academic sense). 
>  The license forces that plus proper academic credit (ie a citation) for 
> the benchmark suite, which personally I think is quite fair given how 
> much work was put into putting it together (much more than the typical 
> academic paper).

Actually I have the same opinion: we need rigourous benchmark suites in 
academics.  I'm not diametrically opposed to using non-free ones, and I 
do understand the value (as I said, I use a non-free suite all the time 
in my work, so politics in practice don't stop me from using non-free 
software (although it's nice to avoid it)).

Still: why can't there be high quality free Java suites for GNU 
Classpath VM's to test against?  Furthermore, why can't academics use 
and distribute free software?  It would seem to me that they're 
particularly in a good position to do so.

> I know some of the people who put together the benchmark suite fairly 
> well and it was a massive effort (think person-years, not person-weeks; 
> good, usable, portable, benchmark suites are a lot of work).  It would 
> be a shame if your politics prevented you from using it, but that's life 
> I suppose.

I don't mean to start a political tirade here, but if you look at all 
the free VM's and all of GNU Classpath, that's required a *much* larger 
effort than putting together one benchmark suite (composed of free or at 
least public domain software at that).  So I don't really see why it 
should be a problem for interested parties to accomplish, although it 
obviously won't happen overnight.

Perhaps a replacement for DaCapo and SPECjvm98 is not in order: this 
would make comparison with other papers in the literature difficult and 
just confuse the issue.  So, how about a good *companion* suite then?

Chris