Re: Test suite and autotest

[Tim Landscheidt]

Paul Smith <address@hidden> wrote:

>> What is the sentiment concerning converting the test suite
>> to autotest?  Attached is an initial, unpolished patch for
>> one test.

> Hi Tim; thanks for taking an interest in this!

> I have concerns about autotest, though I'm not hugely familiar with it.
> Some of the things we need to do with the make test suite, I'm not sure
> how to do  well in autotest.  For example many messages (especially
> errors) that need to be tested by the test suite include values such as
> the name of the make program, the pathname to the current directory,
> etc. none of which can be made static.  The test suite (check
> run_make_tests.pl:run_make_test()) replaces tokens in the makefile
> output to get the right content.

> Also there are more advanced uses like the ability to run the suite
> under valgrind (or other tools in the past but these days I only really
> use valgrind, myself).

> And to be honest, I like the format of the run_make_test() function much
> better than the m4-based format of autotest, although perhaps with some
> macro usage that could be fixed up.

> Still, autotest does provide some nice features including a framework
> with logging etc. and solving the issue of "leaking" information between
> test groups etc.  And, it doesn't require Perl which could be a help (or
> maybe not: is it more portable to require Perl on non-POSIX-based
> systems like Windows, or to require a Bourne shell?)

I stumbled upon GNU's participation in Google's Summer of
Code and found out that I qualify as a student, so you may
read the following with this in mind :-) (also as a caveat
in that this might influence my thoughts by whether some
item fits into a two-month horizon).

  I can think of two possible routes:

- Plan A: Convert the individual tests to individual auto-
  conf's autotest tests.

- Plan B: Parallelize and convert the individual tests to
  output TAP and depending on the progress of autotool's
  "Interfacing with a test protocol like TAP or subunit"
  GSoC project and developers' preference either:

  a) use that,

  b) write a very basic connector between TAP and autotest
     tailored or rather limited to GNU make's needs (i. e.
     no fancy statistics, but more of a "ok" or "fail" func-
     tionality), or

  c) use Test::Harness.

Both plans would result in that the test suite could run in
parallel, it would be written in a standardized form that is
easily accessible to developers not familiar with GNU make,
and it would be usable in a standardized form (i. e. how
many tests are run in parallel, which tests are run, where
are directories created, etc.) that is easily accessible to
users not familiar with GNU make.

  With regard to plan A, what are concerns to you, I see as
opportunities :-).  autotest is a bit bare with regard to
text manipulation, and I think that would be one of the ben-
efits to demonstrate: How to test for variable text, permu-
tations, etc.  Or, as you allude to, how to work around li-
mitations on certain systems.  valgrind & Co. are very easy
to enable for the whole suite thanks to the macros; see Bi-
son's test suite for example.  What I also like about this
approach is that GNU make essentially deals in shell
scripts, and autotest is just that, so there is no unneces-
sary friction.

  Plan B on the other hand is rather straightforward and not
very challenging, but has the advantage that it will proba-
bly cause less issues with target hosts where the current
test suite works, but there is no developer to give feed-
back.

  From a glimpse at the code, parallelizing the current test
suite in its present form would probably involve reading all
and touching *quite* a few lines just to ensure that every
test has its own clean directory, not to speak of actual pa-
rallelization with processes/threads.  And when literally
every stone has to be turned, I'd rather use the opportunity
to go for an existing, established framework.  But I just
want to point out that a) the current test suite works and
b) compared with other projects, its running time isn't very
long and thus cannot be optimized by parallelization very
much :-).

Tim