Re: [Fwd: Re: [avr-gcc-list] GCC-AVR Register optimisations]

[Dave N6NZ]

Andrew Hutchinson wrote:
> Here my input:
>
> For starters gcc has testsuite that can be used. It's not perfect but 
> its quite demanding - even if we cant do all the tests.

Yes, we could probably pull a subset of meaningful test from that which 
would give us a great start.  The hard/tedious part will be generating 
our expected results.

> After than I suggest some "benchmark"  that would produce more normal 
> code and also give qualitative indications of performance (size is easy, 
> speed would be nice).
>
> Finally, regression tests using testcases and bug reports.

Performance regressions are particularly nasty to test. First you have 
to have code that can sensitize an optimization that you want.  Then, 
you need to have some expected results.  The compare of actual to 
expected is difficult -- what are you measuring?  And how close is good 
enough?  Exactly clock count?  Clock count within a guard band?  A 
specific assembler sequence that must happen (but actual registers used 
don't matter)?  Some registers matter?

Then you have the test matrix... what is the expected result with -Os 
versus -O3?  etc, etc, etc.

All this presumes a test framework that can capture all the required 
outputs to check against expected results: .S, clock count, code size, 
exit code,....  Oh, and getting the right numerical (or whatever) 
answer... and are we getting it for the right reason?  The compiler 
might show a great speed and code size improvement by mistakenly 
ignoring the volatile keyword in some optimization, and a simulator 
might never show a wrong answer :(

Having been purely a gcc and avr-gcc user, I don't have any idea if any 
parts of this framework already exist in a form that addresses these 
problems.

Anyway, the test matrix is huge, and a simulator is going to limit the 
ultimate performance of the test rig.  So we won't be able to do 
everything we can think of.

On the plus side of the ledger: Our audience, being embedded developers, 
have very specific needs, so that can help prioritize.

OK, this e-mail was a shotgun blast of issues completely devoid of 
specifics :) :)

> Andy

-dave

> Dave N6NZ wrote:
> > Well, after spamming the wrong list with this, I hope I've got the 
> > right place now :(
> >
> > -dave
> >
> > ------------------------------------------------------------------------
> >
> > Subject:
> > Re: [avr-gcc-list] GCC-AVR Register optimisations
> > From:
> > Dave N6NZ <address@hidden>
> > Date:
> > Thu, 10 Jan 2008 12:35:10 -0800
> > To:
> > gEDA user mailing list <address@hidden>
> >
> > To:
> > gEDA user mailing list <address@hidden>
> >
> >
> >
> >
> > Weddington, Eric wrote:
> > > - I, and others, are very interested in what you are using to test your
> > > proposed changes. I have plans to put together an AVR Benchmark Suite,
> > > consisting of a variety of publicly available programs that can be used
> > > to test the compiler performance over time. It definitely needs to have
> > > different types of programs, and publicly available programs so there
> > > are no issues with distributing such a Benchmark Suite. I welcome any
> > > collaboration on this.
> >
> > Hi,
> >   Over the course of years I have worked in and managed various 
> > different hardware and software validation teams.  This is probably a 
> > good way for me to contribute, since I'm not a compiler back-end guru, 
> > but I have relevant experience writing and reviewing test plans.  Some 
> > of those test plans even targeted C compilers :)
> >
> > -dave
> >
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > AVR-GCC-list mailing list
> > address@hidden
> > http://lists.nongnu.org/mailman/listinfo/avr-gcc-list
> >