Re: [PATCH 1/5] {test-protocols} parallel-tests: make parsing of test results safer

[Ralf Wildenhues]

* Stefano Lattarini wrote on Wed, Jul 20, 2011 at 10:05:14PM CEST:
> Hi Ralf, thanks for the tips.

Rather, 5 minutes of searching the web.  But sure, anytime.  ;-)

> On Tuesday 19 July 2011, Ralf Wildenhues wrote:
> > Test suite environments:
> > any that use the above
> >
> I see now that prove(1), the default command line interface to TAP::*
> modules, has a `--state' option that allow it to store the results of
> the previous testsuite run.  But the exact way this is done seems to
> be an internal detail; should I look into this nonetheless?

Why do you need to ask?  Wasn't one of the first parts of this project
looking into how other test environments work?

> > dejagnu
> >
> After having read the blog post you link below, I'd rather not look into
> this at all ...

Learning from real-world software, including mistakes, can be a lot more
enlightening than from shiny but unused software.

> > qmtest
> >
> Just as an aside: the documentation of this tool seem to employ the same
> terminology we are using in automake w.r.t. test results (in particular,
> it uses "FAIL" and "ERROR" in the same way we do, making clear distinction
> between the two); so we've probably managed to avoid NIH in this respect.

No.  Avoiding NIH means not doing your own thing in the first place,
but doing the research first, and then acting intelligently afterwards.

> > http://en.wikipedia.org/wiki/Portal:Software_Testing lists several

There are probably a dozen more in this web page alone.  I don't think
it is asked too much to at least look around in them in search for
valuable data for all kinds of things (future features you don't want to
prevent being possible by wrong design decisions now).  Even much more
generally than just for the result format.

Please take this as an opportunity, not as a burden.  Sometimes copying
ideas is just the right, the best thing.  And learning from others
almost always is.

> > Autotest
> >
> Hmmm... the recheck target of autotest appears to work through some
> sort of "screen scraping" applied to the testsuite logs (not an example
> I'm eager to follow I must say), and the format of the generated
> `testsuite.log' itself seems quite ad-hoc, "grown" rather than designed.

Sure, hacked that in a few hours, and I'm not proud of it, but it
works reliably enough and was fast enough.

Cheers,
Ralf