Re: [Simulavr-devel] regression testing proposal RFC

[reinhard . jessich]

On Wed, 16 Jan 2002, Theodore Roth wrote:
> Howdy,
> 
> While letting my mind wonder the last few days, I think I've come up with
> a way to attach a regression testing system to simulavr which will require
> no changes to simulavr as it stands now. I feel that a regression testing
> system is important to insure that nothing is broken as new features and
> devices are added to the core.
I like this. I use it in my daily work in my company. We have a target
regression_test  in our Makefiles. This will execute a script Regressiontest.
This script must return a positive result.

We have a commit script, which will commit, checkout in a temp dir and
make regression_test. If all works, it will tag the new release of the package.

> The trick here is to use the existing gdb and display coprocess interfaces
> in a psuedo-feedback loop. Here's an example invokation of what I have in
> mind:
> 
>   $ export SIM_DEVICE=at90s8515
>   $ simulavr -d at90s8515 -g -P regression_test_program
> 
> The SIM_DEVICE env variable is needed by the regression program so that it
> knows explicitly which device is being simulated. Once simulavr has
> started the regression program and is listening for a gdb connection, the
> regression program will connect back to the simulator via the gdb remote
> serial protocol. The regression program will control the sim just like gdb
> would and will automatically obtain register, flash, and memory updates
> via the display interface. All the regression tests for each device will
> be scripts automatically run by the regression program. Test results will
> be written to a file for post processing by a master control
> script/program. I'm considering writing the master control script and the
> regression program in python.
This sounds possible.
It seems that you like python, I like perl, but I don't like to start war. I
think I can learn python, if you prefer it.

> I'd like to add this during the 0.1.x series. Setting up the
> infrastructure should only take me about a week, but the actual test cases
> will take a while. For 0.1.x, I think the test cases for all the opcode
> handlers should be enough work and I may ask for others to help with
> writing some of those.
My experience is, that it is better to have some scripts, which will check a
few things, than no regression test. If you provide the master control script
and some major test scripts, I am sure there are some people which will provide
new test cases.

> I haven't done any formal design yet, I just wanted to get my idea written
> down and open a discussion.
> 
> Comments and criticisms?

Regards,
   Reinhard

-- 
 Ing. Reinhard Jessich              mailto: address@hidden
 A-1190 Vienna, Goergengasse 2/2/1  phone: +43/1/3692600
 http://members.telering.at/jessich mobile: +43/664/1735439