[Axiom-developer] Axiom test suite

[daly]

David,

Axiom has a rather extensive test suite. See the Computer Algebra
Test Suite (CATS) on this page:
http://axiom-developer.org/axiom-website/CATS/index.html

You can build Axiom and run various levels of testing by
including TESTSET=option on the make command line. The 
options are

  TESTSET=notests (the default)

Axiom is tested before release. There is no real need to run
the tests during a user build. 

  TESTSET=regresstests (tests in src/input)

The src/input directory contains a random collection of examples
picked up over time. They are being reorganized so that specific
tests are put in the algebra books and more general pamphlets
are developed into more extensive tutorials. Note that some of
these files explicitly cause failures in order to test failure 
mechanisms.

  TESTSET=algebratests (tests in books/bookvol10.*)

The algebra books (10.2 Category, 10.3 Domain, 10.4 Package,
10.5 Numerics) contain input files for each specific piece
of algebra. This will extract those files and run them.

  TESTSET=catstests (schaum integration and kamke diffeq)

The catstests cover the Schaums book of integrals and the Kamke
book on differential equations.

  TESTSET=richtests (Albert Rich test suite)

The richtests suite is a (growing) portion of the 50,000 
integrals published by Albert Rich.

  TESTSET=newrichtests (long running integration test suite)

Some of the Rich integrals take a LONG time to compute. The
future plan is to fork Axiom and run the test suites in parallel.
Until then, these are separate.

  TESTSET=alltests

Be aware that 'alltests' take about a day to run on my fastest
machine.

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

THE TEST MACHINERY

The test machinery is trivial. Each test suite has the expected
result as specially-tagged comment lines (--R) enclosed in 
numbered comment lines (--S, --E). So in "foo.input.pamphlet"
you'll see blocks like:

--S 17 of 59
x^2
--R
--R        2
--R  (17) x
--E 17

The --S, --R, and --E lines are valid Axiom comments.
When this file is run it creates "foo.output" containing the
actual result as well as the comment lines:

--S 17 of 59
x^2

        2
  (17) x
                                         Type: Polynomial(Integer)
--R
--R        2
--R  (17) x
--R                                         Type: Polynomial(Integer)
--E 17

There is a function called "regress" which reads "foo.output",
compares the actual result with the expected result and outputs
PASS or FAILED for each test. It also complains if there are any
missing or unfinished tests (we told it there were 59 above).

Because the test files are little more than console logs it makes
creating a test file trivial. Just run your input file, edit the
console log by adding the --S, --R, and --E comments and you have
a valid test set. Wrap a few latex lines (e.g. copied from 
algaggr) around it and you have a valid pamphlet file for testing.

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

RUNNING TESTS BY HAND

Axiom test files are vanilla latex files (called pamphlet files
for historical reasons). Axiom knows how to extract "chunks"
from latex files using the )tangle command.

1) pick a file from src/input (e.g. algaggr.input.pamhlet)
2) at the Axiom command line type:

   )cd src/input        -- go to where the file is
   )tangle algaggr      -- extract the file algaggr.input
   )read algaggr        -- run it, creating algaggr.output and exit

3) restart Axiom

   )regress algaggr     -- compare the actual vs comment lines


Hope this helps.

Tim