Re: [Quilt-dev] test suite improvements

[Jean Delvare]

Hi again John,

(Note that I didn't have the opportunity to look at your patches, I'm
commenting on ideas rather than implementation.)

> 1. I like to build test cases for bugs, and then layer a bug fix on
> top of that, so the repercussions of the bug fix can be seen.

Technically, what shows the repercussions of a bug fix are the other
tests of the suite. Making sure that the fix worked is usually easy,
this could even be done manually in most cases. Making sure that the
fix has no nasty side effects is what the test suite is good for.

That being said, building a test case locally before fixing a bug is an
excellent habit.

> I would like to take this one step further, and record that the
> recorded output is incorrect.  Instead of prefixing the line of output
> with >, undesirable output could use ? instead.  test cases for bugs
> can then be committed as soon as they are reproducible, found using
> grep, and worked on at a later date.
> 
> Does anyone object to having the bug test case being committed prior
> to the bug fix?

What good would it do? I see little interest in recording known
failures in the test suite. If you can't fix the problem now, better
add an entry to the TODO list, detailing the failure that was observed,
on what system, and any preliminary analysis that was done at this
time. Maybe you can add a pointer from the failed test file to the TODO
list in the form of a comment if you really think people will look for
additional information in the test file rather than the TODO list.

> 2.1 regex support for output lines that commence with >~

Sounds good to me.

> 2.2 failed commands to use the separators !~, =~, != and == instead of | and ?

Not sure I get what you mean here; unneeded at first sight.

> 3. local-quilt-x.diff introduces a 'make check'.  This becomes very
> important as more of the ./configure information becomes necessary to
> accurately test the quilt built in the local workpit.  Is it ok to
> remove test/Makefile?  The only alternative that I can see is to
> replace it with test/Makefile.in.

If this is needed, sure, no objection.

> 4. regularly running all of the tests on msys is really slow.  I would
> like to record which tests have run to completion, and pick up the
> testing on the last test case to fail.  I wrote a rough patch to do
> this last year; any suggestions on how this should be done in order to
> be committed?

Good idea. The test suite is relatively slow even under Linux on my
aging laptop.

Maybe we can have it start over automatically if patchfns is found to
have changed since the checkpoint files were written? So that people
don't think they passed the test suite if they changed that common file
to fix a problem in the middle of the suite, as such a change may break
earlier tests. Oh well, just a random thought anyway.

> 5. Also, when one command in a test case fails, the rest are sure to
> follow.  I would like to change the test harness so that it only
> continues past the first error if an option (--ignore-error) is
> enabled.  I think this behaviour should be disabled by default.

+1. I've been thinking the very same many times before, but never
took the time to actually implement a fix.

Thanks,
-- 
Jean Delvare