[Axiom-developer] AxiomUnit, DejaGnu, and QMTest

[Bill Page]

The Axiom distributioin has a set of .input files that are run as
"regression tests" during the Axiom build, unfortunately there is
currently no facility for automatically checking the results of
these tests. Gaby has suggested DejaGnu and QMTest as possible
tools for automatically running these tests.

See:

http://www.gnu.org/software/dejagnu
http://www.codesourcery.com/qmtest

Ralf and Martin mentioned AldorUnit as possibly relevant and suggest
that AldorUnit could be re-implemented in SPAD for Axiom (AxiomUnit).
See:

http://www.risc.uni-linz.ac.at/software/aldor/project.php?id=AldorUnit

To better understand the issue I think we need to distinguish between
System Testing and Unit Testing:

http://en.wikipedia.org/wiki/Unit_testing
http://en.wikipedia.org/wiki/System_testing

>From my point of view the primary purpose of the Axiom .input files
and the DejaGnu and QMTest tools is System Testing. To this end,
these tools drive the complete system using a script and comparing
the computed results to the expected results.

Unit testing on the other hand is (usually) and integral part of
the development and usually consists of testing some module or
subroutine locally and in isolation from other units. AldorUnit is
an example of an application program interface (API) intended to
facilitate such tests. Typically one might write unit tests as
additional optional debugging code that tests for expected conditions
(assertions). The testing of these assertions might be enabled or
disabled at run-time or at build-time. Because the assertions are
checks only and are not needed to produce the desired results, they
are usually disabled when running the software in a "production"
mode. As far as I can see currently Axiom development does not
currently employ any unit tests in an essential way.

Unit testing and system testing are not alternatives but rather
should be used together at different stages in software development
and testing.

Regression testing is a part of both Unit and System tests:

http://en.wikipedia.org/wiki/Regression_testing

DejuGnu and QMTest support automated regression testing at the
System level.

DejuGnu also has the ability to drive unit tests through an API
for C++. It is possible to imagine that AxiomUnit could be written
in such as way that it could also be called within DejuGnu for
such tests.

As for objections to DejuGnu and QMTest, I see the following as
possibly significant:

1) DejuGnu depends on TCL this would introduce another dependency
   in the Axiom build environment.

2) Similarly QMTest is written in Python, which would also be a
   new dependency.

Personally I am not concerned about either of these issues.

As for choosing one over the other, QMTest has a nice web browser-
based graphical user interface and is said to work equally well on
Windows, MAC and Linux systems, while DejaGnu is possibly more
mature and integrates directly with the automake/autoconf tools.

Regards,
Bill Page.

On November 18, 2006 2:40 PM Gaby wrote:
> ...
> > 
> > This ultimately raised the issue of how to enforce testsuite
> > and the whole build process.  I don't have much time left this
> > afternoon. I know of DejaGnu framework and the QMTest framework
> > for running testsuite.  I have far more extensive experience
> > with DejaGnu than with QMTest.  On the other hand, I'm pretty
> > sure people would object to DejaGnu for various reasons.
> > 

On November 18, 2006 3:55 PM Bill Page wrote:
> 
> What reasons?
> 
> I took a quick look at DejaGnu.
> ... 
> I did not see anything to which I would immediately object.
> 

On November 18, 2006 3:50 PM Ralf Hemmecke wrote:
> ... 
> Sorry, I don't know any of them, but for the Algebra part it might
> be possible to rewrite AldorUnit so that it could be used with Axiom
> and SPAD instead of Aldor.
> 
> http://www.risc.uni-linz.ac.at/software/aldor/
> 
> I could imagine that the rewrite should not be terribly hard.
> 

On November 18, 2006 3:50 PM Martin Rubey wrote:
> 
> I started to do it, no it doesn't seem to be hard. The main 
> trouble is that we
> would need to implement TextWriter, but that might be a good thing
> anyway. Although I must say that I like OutputForm.
> 
> Unfortunately, the compiler segfaults over my modifications currently.
>