RE: [pooma-dev] RFA: Proposed Modifications to Test Files

[Allan Stokes]

<<<
Once upon a time we had a testing team and under their auspices
we formed the r2/test directory. This was supposed to test things based on
specifications; i.e. it was supposed to be somewhat black-box, whereas the
unit tests (src/Engine/tests, etc) were written by the developers and were
pretty much white-box. This process did not work out (we spent more time
debugging the test-team's test codes than we did debugging the things they
were testing).
>>>

Jim's comments mirror my experience with other projects.

IIRC back in the early eighties when the Mathematica/Maple kernels were
still under intense development, it was fashionable to validate huge chunks
of the implementation by computing difficult Ricci tensors for thousands of
hours.

This was surprisingly effective at catching small coding discrepancies which
escaped many other forms of testing.  You can go a long way with a couple of
intense application level tests if they are chosen with good internal
structure.

The first problem with black-box testing at the specification level is that
specifications rarely succeed in defining the problem domain without making
unwarranted incursions into the implementation domain.

A second problem is that specifications rarely map directly onto test cases
anyway.  For example, the specification for a reference counted allocated
will have the invariant that the reference count is equal to the number of
live objects holding an external reference, but in practical terms this
invariant is generally inaccessible to direct inspection.

Finally, black box testing often fails to find problems even in the cases
where the criteria really are self evident.  A good example of this is the
Intel FDIV bug.  Intel's ultimate response was to put more effort into
ensuring that their white-box testing covered the implementation space.

I agree with the comment raised at the Proximation meeting that if people
are dilligent about posting unit tests with each new feature or bug fix most
issues can be tracked more effectively on the ground floor.

I'll hold off reviewing these changes until I hear it decreed that test
remains a going concern.

Allan