Re: @example's as doctests

[Oliver Heimlich]

Am 27.03.2015 um 10:39 schrieb Colin Macdonald:
> Oliver (and octave-maintainers),
>
> I've hacked doctest-for-matlab enough [1] to run it on the infsup class
> from your Interval package.  Interval uses the standard Octave @result{}
> convention: in my patched version, I split on @result{} and some other
> heuristics.  It could be improved, see last bullet below.
>
> [1] https://github.com/catch22/doctest-for-matlab/pull/10
>
> I attach a log: most tests pass.  Oliver: I'm impressed!

Colin,

thanks for sharing the results. The interval package probably is an easy 
target for your project. Most examples are syntactically very similar 
and not very complicated. Any output from this package should be system 
independent. Additionally the @example blocks in the package did not 
have a lot of time to age and become wrong. ;-)

> I noticed:
>    * there are a few actual errors.

I found only one error with ldivide. The documentation string calls 
mldivide by accident (bug 1) and your test happens to reveal a problem 
in mldivide (bug 2). Lucky strike!

>    * there are some cases of @example used as a LaTeX alternative
>      (is there an @verbatim instead?)

Yes, the @example blocks have been used for ASCII art and do not contain 
actual examples. The @verbatim should be a valid replacement (without 
indentation). I am going to change the documentation.

>    * there a few minor (cosmetic?) errors: [-1, 5] to [-1, +5]

I would consider these actual errors and am going to fix the 
documentation. A very early version of the package had a different 
format for the output of signs.

>    * there are a couple cases were my current splitting
>      input/output based on @result{} fails.  (Personally I prefer
>      marking input with ">>" but the doctest code could instead be
>      refactored further.)

I found several problems with doctest:

intervaltotext: The @example block contains 3 consecutive inputs and 
outputs. Without the missing “>>” for inputs there is a problem with 
seperating them. Would it be sufficient to use three @group blocks in 
the @example block?

eq, nextout: The output is boolean, but the documentation string shows 
the console output as ans = 0 (or 1). I can change the documentation to 
@result{} False (or True).

mulrev: The function returns two return values and the syntax used in 
the @example is probably wrong for that purpose.

> Finally. I checked several functions in core: spdiags and svd are fairly
> close to testable.  qr, chol not really written with this sort of tool
> in mind.  IIRC, others warned me this was true of core Octave.
>
> Colin

I plan to improve the performance of console output for my package. I 
can put your work into great use during regression testing for that change.

Following our discussion in February, I have implemented all @example 
blocks in the package as %!test blocks. They check the return value 
semantically and can be run by any user. However, doctest also checks 
the @example blocks syntactically, which provides additional value. And 
as the ldivide example tought us: The %!test blocks are not guaranteed 
to be perfectly synchronized with their @example blocks!

Oliver


> On 02/02/15 18:23, Oliver Heimlich wrote:
> > Am 02.02.2015 um 18:42 schrieb Colin Macdonald:
> > > I've recently learned how to use address@hidden to put the expected result
> > > into the texinfo docs of my functions.  (Thanks to "interval" pkg dev!)
> > >
> > > Is there any existing way to run these as tests?  Like doctest in
> > > Python?  I.e., extract them, run them, make sure the output matches the
> > > address@hidden  The documentation refers to "example" but that seems to
> > > pull in `%!demo` blocks instead.
> >
> > The @result{} is just a macro that produces the ⇒ symbol in the
> > documentation. I find it useful for short examples in the documentation.
> > I copy the console output into the @example, replacing “$variable = ”
> > with the macro.
> >
> > As a package maintainer I prefer to have these examples as test cases,
> > just to make sure that the documentation is correct. That's why I like
> > your proposal. The simplest solution IMHO is to add a %!test block for
> > examples used in the documentation manually.
> >
> > It is complicated to check the @examples automatically, you must
> > consider:
> >  - The @example blocks must not follow any particular syntax, nor must
> > they contain useful code,
> >  - the author of the example blocks may have abreviated commands or
> > console output,
> >  - if you wanted to check the console output of the examples, you would
> > have to intercept the stdout,
> >  - some results may be system-dependent,
> >  - …
> >
> > > I expect the answer will be "no, but please write such a thing"... :(
> >
> > I'd find it useful to code simple %!test blocks, which will be used to
> > generate additional @example blocks at the end of the documentation.
> > Perhaps introduce a new %!example block?! However, automating this is
> > probably not worth it, since this only makes sense for short code
> > snippets and the examples in the documentation usually do not change
> > often.