AVR Benchmark Test Suite [was: RE: [avr-gcc-list] GCC-AVR Register optimisations]

[Weddington, Eric]

> -----Original Message-----
> From: Dave N6NZ [mailto:address@hidden 
> Sent: Thursday, January 10, 2008 1:35 PM
> To: gEDA user mailing list (by accident)
> Subject: Re: [avr-gcc-list] GCC-AVR Register optimisations
> 
> 
> Hi,
>    Over the course of years I have worked in and managed various 
> different hardware and software validation teams.  This is probably a 
> good way for me to contribute, since I'm not a compiler 
> back-end guru, 
> but I have relevant experience writing and reviewing test 
> plans.  Some 
> of those test plans even targeted C compilers :)
> 
> -dave
>  

> -----Original Message-----
> From: 
> address@hidden 
> [mailto:address@hidden
> org] On Behalf Of John Regehr
> Sent: Thursday, January 10, 2008 4:34 PM
> To: address@hidden
> Subject: RE: [avr-gcc-list] GCC-AVR Register optimisations
> 
> > - I, and others, are very interested in what you are using 
> to test your
> > proposed changes. I have plans to put together an AVR 
> Benchmark Suite,
> > consisting of a variety of publicly available programs that 
> can be used
> > to test the compiler performance over time. It definitely 
> needs to have
> > different types of programs, and publicly available 
> programs so there
> > are no issues with distributing such a Benchmark Suite. I 
> welcome any
> > collaboration on this.
> 
> This is excellent and much needed, Eric.
> 
> I can help with some TinyOS applications.
> 
> Hopefully the benchmarks can be packaged up with a simulator 
> in order to 
> get dynamic information as well as static.  I use Avrora 
> (first google hit 
> for "avrora") for everything, though it supports relatively few chips.
> 

Hi John, Dave, others,

Here are some random thoughts about a benchmark test suite:

- GCC has a page on benchmarks:
<http://gcc.gnu.org/benchmarks/>
However all of those are geared towards larger processors and host
systems. There is a link to a benchmark that focuses on code size,
CSiBE, <http://www.inf.u-szeged.hu/csibe/>. Again, that benchmark is
geared towards larger processors.

This creates a need to have a benchmark that is geared towards 8-bit
microcontroller environments in general, and specifically for the AVR.

What would we like to test?

Code size for sure. Everyone always seems to be interested in code size.
There is an interest in seeing how the GCC compiler performs from one
version to the next, to see if optimizations have improved or if they
have regressed.

There is also an interest in comparing AVR compilers, such as how GCC
compares to IAR, Codevision or ImageCraft compilers.

And sometimes there is an interest in comparing AVR against other
microcontrollers, notably Microchip's PIC and TI's MSP430.

Because there are these different interests, it is challenging to come
up with appropriate code samples to showcase and benchmark these
different issues. But we could also implement this in stages, and focus
on AVR-specific code, and GCC-specific AVR code at that.

If we are going to put together a benchmark test suite, like others
benchmarks for GCC (for larger processors), then I would think that it
would be better to model it somewhat after those other benchmarks. I see
that they tend to use publicly available code, and a variety of
different types of applications. We should have something similar. Some
suggested projects: FreeRTOS (for the AVR), uIP (however, we need to
pick a specific implementation of it for the AVR; I have a copy of
uIP-Crumb644), the Atmel 802.15.4 MAC, and the GCC version of the
Butterfly firmware. I also have a copy of the "TI Competitive
Benchmark", which they, and other semiconductor companies, have used to
do comparisons between processors. I also have a copy of some Atmel
internal AES code. I believe that this code is publicly available in
some kits that Atmel offers, but I need to do some double-checking to
make sure that it is public. 

John, I would welcome publicly available code from TinyOS, but I would
need to be already compiled with nesc, so that way we just have straight
C that we can feed into avr-gcc. I've been aware of avrora for several
years. If it is useful for this kind of work, I'm open minded to it. I
just want to make sure that it works and is somewhat easy to use.

Does anyone have other suggestions on projects to include in the
Benchmark? One are that seems to be lacking is some application that
uses floating point. Any help to find some application in this area
would be much appreciated.

There needs to be some consensus on what we measure, how we measure it,
what output files we want generated, and hopefully some way to
automatically generate composite results. I'm certainly open to anything
in this area. I would think that we need to be as open as possible on
this, with documentation (minimal, it can be a text file) on what are
our methods, how the results were arrived at, but importantly that the
secondary/generated files be available for others to review and verify
the results.

On practicalities: I am certainly willing to host the benchmark test
suite on the WinAVR project on SourceForge and use it's CVS repository.
If it is desired to have it in a more neutral place, such as avr-libc,
I'm open to that too, if Joerg Wunsch is willing.

Thoughts?

Thanks,
Eric Weddington