[Quilt-dev] test suite improvements

[John Vandenberg]

Hi again,

In order to complete the portability series that I have been working
on, there are a few improvements that I would like to propose for the
test harness/suite.

1. I like to build test cases for bugs, and then layer a bug fix on
top of that, so the repercussions of the bug fix can be seen.

I would like to take this one step further, and record that the
recorded output is incorrect.  Instead of prefixing the line of output
with >, undesirable output could use ? instead.  test cases for bugs
can then be committed as soon as they are reproducible, found using
grep, and worked on at a later date.

Does anyone object to having the bug test case being committed prior
to the bug fix?

2. As part of my original attempt to port quilt to msys, I found a
number of binaries were producing slightly different messages to those
in the test cases.  The worst offender was patch-2.5.  While it would
seem a little anal to try to attempt to support such a historical
release, it works fine and msys is a pretty finicky platform to
maintain packages for, so I dont expect a new version of patch being
able to compile on it for quite a while.  Also, I am sure that
sometime in the future, a binary that quilt uses will decide to emit a
slightly different error message, and it will break the test cases.

In order to address this, I propose the following changes to the test harness:

2.1 regex support for output lines that commence with >~
2.2 failed commands to use the separators !~, =~, != and == instead of | and ?

3. local-quilt-x.diff introduces a 'make check'.  This becomes very
important as more of the ./configure information becomes necessary to
accurately test the quilt built in the local workpit.  Is it ok to
remove test/Makefile?  The only alternative that I can see is to
replace it with test/Makefile.in.

4. regularly running all of the tests on msys is really slow.  I would
like to record which tests have run to completion, and pick up the
testing on the last test case to fail.  I wrote a rough patch to do
this last year; any suggestions on how this should be done in order to
be committed?

http://zeroj.hda0.net/quilt/patches/test-make-depend.diff

5. Also, when one command in a test case fails, the rest are sure to
follow.  I would like to change the test harness so that it only
continues past the first error if an option (--ignore-error) is
enabled.  I think this behaviour should be disabled by default.

--
John