Re: Benchmarks

[shudo]

Andrew and Dalibor,

> Andrew Haley wrote:
> > What do people use for benchmarking their JVMs?  I know about
> > CaffeineMark, but ere there any other good microbenchmarks that are
> > popular?
>
> Hmm, microbenchmarks are hard, I think there are many left over, since 
> HotSpot like engines don't really get a fair result on microbenchmarks [1].

I have seen very informative slides on microbenchmarks:
TS-1816: How NOT To Write A Microbenchmark.
They can be found on the web pages listing talks in JavaOne 2002:

  JavaOne[sf2002] - Conference - Sessions
  
http://servlet.java.sun.com/javaone/sf2002/conf/sessions/all-all-regular.en.jsp

According to the slides, even SPEC JVM98 has numerous defects.
I could also find problems of the benchmark suite and
mentioned them in the following web page:

  Performance comparison of JITs (Jan 2002)
  http://www.shudo.net/jit/perf/

  But ORP and Jikes RVM cannot run a Java applet and Kaffe cannot run
  the SPEC tools as an applet loaded from a web server. _201_compress
  and _213_jack benchmarks are very sensitive to where the benchmark
  programs are loaded from. Thus the results of these two benchmarks
  cannot be compared fairly among all runtime systems.

The above web page has results of some benchmarks such as
SPEC JVM98, SciMark 2.0, Linpack benchmark, Eratosthenes Sieve and
CaffeineMark 3.0.

I want to update the performance comparison page because many Java
runtimes have been updated since the last benchmarking (Jan 2002) and
now I have several interesting computers having Itanium 2, Opteron and
PowerPC 970 (yes, it's G5).  I hope I can have a time to do that.

  Kazuyuki Shudo        address@hidden  http://www.shudo.net/