Simulator for GCC Testing [was: RE: [Fwd: Re: [avr-gcc-list] GCC-AVR Register optimisations]]

[Weddington, Eric]

> -----Original Message-----
> From: 
> address@hidden 
> [mailto:address@hidden
> org] On Behalf Of Andrew Hutchinson
> Sent: Thursday, January 10, 2008 6:27 PM
> To: address@hidden
> Cc: address@hidden
> Subject: Re: [Fwd: Re: [avr-gcc-list] GCC-AVR Register optimisations]
> 
> Here my input:
> 
> For starters gcc has testsuite that can be used. It's not perfect but 
> its quite demanding - even if we cant do all the tests.
> 
> Do we have info on setting this up with simulator? I did have some 
> instruction - once!
> 
> After than I suggest some "benchmark"  that would produce more normal 
> code and also give qualitative indications of performance 
> (size is easy, 
> speed would be nice).
> 
> Finally, regression tests using testcases and bug reports.
> 

Hi All,

Some points:

- Yes, GCC does have a Regression Test Suite, and it can execute for the
AVR using the SimulAVR simulator. There are many, many tests that pass
for the AVR. There are quite a few that don't, but most of those
failures that I have looked at either the test needs fixing (because it
assumes a 32-bit processor), or the tests don't apply to the AVR. Some
work needs to be done to get the Regression Test Suite in shape for the
AVR.

- As mentioned, simulavr is known to work as a simulator for the GCC
test suite. However, simulavr is not really maintained anymore. At the
simulavr project on Savannah, there is a new code base called simulavrxx
which is based on C++. This is maintained, but it could use help: It
doesn't run on Cygwin yet, and AFAIK it cannot run the GCC Test Suite
yet. Any help on this is deeply appreciated.
<https://savannah.nongnu.org/projects/simulavr>

I strongly recommend that the wheel not be reinvented. If people are
interested in running the GCC Regression Test Suite, I would recommend
using available tools, and improving the available tools rather then
invent new ones.

I have instructions on running the GCC Regression Test Suite (from
Bjoern Haase, IIRC). I have yet to run it myself, but others have done
so successfully. However, there are reports about difficulties on
running the test on Cygwin. I have heard that it is successful on Linux.
There is a person from Belgium, Mike Stein, who has been running the GCC
Test Suite for the AVR pretty much on a daily basis and he has been
posting the results regularly on the gcc-testresults mailing list:
<http://gcc.gnu.org/ml/gcc-testresults/>. Just search for "avr" to see
the results. (It looks like he last did it in December.) I'll be
attempting to run the GCC Test Suite probably sometime in Q1 2008. (I'm
going to be busy in January and February.) Just email me if anyone is
interested in the instructions.

Note that running the GCC Test Suite is imperative for anyone who works
on GCC, because in order to submit any patches to the GCC project, they
require that the patch is tested with the Test Suite and that there are
no new regressions. That's the main purpose of the test suite.

I'll start a new thread about a benchmark suite...

Thanks,
Eric Weddington