Re: benchmarks (was Re: Progress on a Classpath mauve suite?)

[Per Bothner]

David P Grove wrote:
> I think you need to understand the point of the benchmark suite.  The 
> whole goal is reproducible science, so if someone doesn't cite the exact 
> version of the benchmarks, then it isn't useful (in an academic sense).

That can be achieved without the DaCapo restrictions.  For example TeX
has fairly strict requirements on naming and versioning.

But DaCapo license is extremely restrictive.  It's not clear that a
company could use DaCapo to benchmark their products - even open-source
products, since that might be defined as "for-profit".

Forbidding one to "alter, modify, improve, decompile, disassemble or otherwise
reverse-engineer" has nothing to do with "reproducible science", and in
fact is contrary to acadmic openness and "reproducible sciece".

>  The license forces that plus proper academic credit (ie a citation) for 
> the benchmark suite, which personally I think is quite fair given how 
> much work was put into putting it together (much more than the typical 
> academic paper).

A requirement for credit isn't inherently incompatible with Free Software:
The old BSD license with the "notorious advertising clause" may have
been awkward and GPL-incompatible, but it was accepted as Free.

> I know some of the people who put together the benchmark suite fairly 
> well and it was a massive effort (think person-years, not person-weeks; 
> good, usable, portable, benchmark suites are a lot of work).  It would 
> be a shame if your politics prevented you from using it, but that's life 
> I suppose.

You're rather missing the point, methinks.  I know some of the people
at Redmond who've spent person-decades to put together quite a nice operating
system.  I also know some people in Santa Clara who've spent person-decades
putting together a very nice Java implementation.  It's a shame that
politics are preventing us from using either.
--
        --Per Bothner
address@hidden   http://per.bothner.com/