Is Autotest the right solution?

[Nathaniel D. Sizemore]

Autoconf developers,

My organization develops scientific software.  Currently for  
regression testing we have a set of custom scripts that build the  
source code nightly, run a series of simulations, compare the  
resulting data with accepted values, and email the status of these  
tests to the development team.  We recently decided to refactor this  
setup to improve ease of use and functionality.  It was decided to  
see if Autotest could be used for this refactoring.

I chose Autotest for this project for several reasons.  We already  
use Autoconf, Automake, etc. extensively in the company, so Autotest  
would be using our existing expertise.  Also, using an existing,  
actively developed OSS project seemed better than continuing to  
maintain and update our from-scratch solution.  However, as we've  
worked through the project, we are reconsidering the choice due to  
feature requirements.

Specifically, the test suite needs to do the following:

1) Run nightly as a cron job to test the entire package
2) Allow developers to easily create and and new tests
3) Serve as an example of running the software for end-users

Unfortunately, some of the requested features have proven difficult  
to implement using Autotest, particularly that of being an example to  
end-users.  The issues I've encountered include:

1) Naming of test subdirectories.  Currently, Autotest runs each test  
in a separate directory; these are labeled sequentially based on the  
number of the test, i.e. 1/ 2/ 3/ and so on.  The dev team would much  
rather see them named after the name of the test, i.e. foo/ bar/ test1/

2) Hierarchy of tests.  Currently, tests are grouped into  
subdirectories by function.  We would like them to be executed in  
directories named in the same fashion, i.e. testsuite.dir/foo/ 
fooTest1/, testsuite.dir/foo/fooTest2, testsuite.dir/bar/barTest1, as  
opposed to testsuite.dir/1/, testsuite.dir/2, etc.

3) Serving as example to users.  In addition to having cron test the  
software nightly, we want the tests to show novice users how to use  
the software.  Due to the complicated nature of the code, this  
involves several steps, including creating an input file, running the  
software itself (using some scripts to deal with serial vs. parallel  
machines, etc.), and processing the results in some fashion (usually  
by calling external applications).  Currently, I do these steps in a  
series of AT_CHECK macros, but these are difficult to read for the  
average user or developer.  Additionally, users need a script that  
they can run completely independently of the regression tests, unlike  
the 'run' script created by Autotest that is tied into the test  
suite.  In other words, if a user is working on a problem of type  
foo, we'd like to point them to the regression test foo, which would  
show them how they should go about setting up and running their problem.

4) Developing in testsuite.dir.  Developers have requested the  
ability to debug and develop new tests in testsuite.dir itself.  That  
is, editing an input file, and then re-running that single test  
again.  Currently, the solution is to have an AT_CHECK command that  
creates a symlink to the input file so it can be edited.  If the test  
fails, the symlink allows a developer to edit the input, and they can  
then re-run the test with the 'run' script created by Autotest.   
However, this removes and re-creates the directory, meaning they have  
to do a 'cd .' to refresh the filesystem handle.  My dev team hated  
this.

With the exception of the directory/hierarchy naming, I think all of  
these problems can be worked around, mostly by creating shell  
scripts.  These would be user-readable and called by Autotest.   
However, doing that I lose benefits of using Autotest in the first  
place, that of testing the return code of each step, being able to  
run extra commands on success or failure, and auto-generated,  
detailed log files.

The main issues seem to be centered around the tests serving as  
examples for the user.  As a tester/maintainer, I find that Autotest  
works fine for my needs.  However, it doesn't work the way the dev  
team would like it to in terms of developing new tests or debugging  
test failures due to the issues above.  So, I suppose my question  
really boils down to, "Is Autotest the right solution?"

Thanks for any and all feedback, comments, and suggestions.

--
Nathaniel D. Sizemore, Technical Staff
Tech-X Corporation | 5621 Arapahoe Ave Suite A | Boulder, CO  80303
E: address@hidden  P: (720) 974-1846  W: http://www.txcorp.com