Re: benchmarks (was Re: Progress on a Classpath mauve suite?)

[Stephen Crawley]

Dalibor Topic <address@hidden>:
> David P Grove wrote:
> > The license forces that plus proper academic credit (ie a citation) for 
> > the benchmark suite, which personally I think is quite fair given how much 
> > work was put into putting it together (much more than the typical academic 
> > paper). 
> 
> I agree that academic credit is very important. Again, I belive that to 
> be a part of proper academic conduct, rather than something that needs 
> to be explicitely enforced in a software license. 

All good and well, but who ensures that people obey the principals
of good academic conduct?  Consider:

  * The results may be published in a journal, etc whose referees and
    editors do not understand the importance of citing benchmark suites
    properly.

  * The results may be published in an unrefereed report or paper ...
    or on some random blogger's website.

  * The results may be published by some person (or company) who is not  
    obliged to follow "academic conduct" principles.  Indeed, it is
    common knowledge that some people/companies are prepared to "cheat" 
    in order to make their product give impressive benchmark results.

Furthermore, I'm not convinced the general "principles of good academic
conduct" really cover this case.  I did a brief google search on
academic conduct and citation, and nothing I found covers this case.
Nowhere did I find a general requirement to cite the source and version
of software used in research, or a particular requirement for
benchmarks. IMO, a benchmark suite is a "process for eliciting data"
rather than a "source of information".

(Note: I'm not saying that one shouldn't cite benchmark software
properly. But, IMO, the primary reason is to this is to allow the
benchmark results to be independently reproduced.)

> Lawyers can't fix bad science. :)

In this case, lawyers (or the prospect of lawyers) CAN fix bad science.

I'm not saying that there isn't a better alternative than a License
agreement.  But relying on nebulous "academic conduct principles" that
are not enforcable and may not even apply is NOT going to work.

-- Steve