[Simulavr-devel] regression testing proposal RFC

[Theodore Roth]

Howdy,

While letting my mind wonder the last few days, I think I've come up with
a way to attach a regression testing system to simulavr which will require
no changes to simulavr as it stands now. I feel that a regression testing
system is important to insure that nothing is broken as new features and
devices are added to the core.

The trick here is to use the existing gdb and display coprocess interfaces
in a psuedo-feedback loop. Here's an example invokation of what I have in
mind:

  $ export SIM_DEVICE=at90s8515
  $ simulavr -d at90s8515 -g -P regression_test_program

The SIM_DEVICE env variable is needed by the regression program so that it
knows explicitly which device is being simulated. Once simulavr has
started the regression program and is listening for a gdb connection, the
regression program will connect back to the simulator via the gdb remote
serial protocol. The regression program will control the sim just like gdb
would and will automatically obtain register, flash, and memory updates
via the display interface. All the regression tests for each device will
be scripts automatically run by the regression program. Test results will
be written to a file for post processing by a master control
script/program. I'm considering writing the master control script and the
regression program in python.

I'd like to add this during the 0.1.x series. Setting up the
infrastructure should only take me about a week, but the actual test cases
will take a while. For 0.1.x, I think the test cases for all the opcode
handlers should be enough work and I may ask for others to help with
writing some of those.

I haven't done any formal design yet, I just wanted to get my idea written
down and open a discussion.

Comments and criticisms?

Ted