Re: Running ert tests on buffers in rst.el and elsewhere

[Stefan Merten]

Hi Christian!

I read your mail just now.

2 days ago Christian Ohler wrote:
> Stefan Merten, 2012-06-18:
>> I wanted to test functions which operate on buffers - i.e. use buffer
>> content as input and possibly modify the buffer. Buffer includes point
>> and mark in this case. Since I found no support for this requirement
>> in the ert package I wrote some support code for it.
> 
> I only glanced at your code so far, but please do take a look at the
> current version of ERT, in particular the utility functions in
> ert-x.el and the way ert-x-tests.el uses them.

I just did. Well, I probably did not give enough documentation in the
first place to understand what ert-buffer.el should be good for: Give
the user an *easy* way to define input buffer and expected output
buffer for a test. That's what `ert-equal-buffer' and
`ert-equal-buffer-return' is for. Part of this is that the user can
give strings describing the buffer so you can write comprehensible
tests like

      (should (ert-equal-buffer (insert "foo") "address@hidden" "\^?foo\^@"))

Once you understood that ?\^@ stands for point and ?\^? stands for
mark the meaning of the test is immediately clear. This supports the
idea that a test also serves as a documentation for the respective
function.

I find no support for this idea in ert-x.el but I may miss something.

> Some preliminary notes:
> 
> Is it true that your code covers two features, one about turning
> buffer content, point and mark into a data structure and back as well
> as comparing such data structures,

Yes - in a preliminary way though.

> the other about providing input to
> interactive functions like completing-read?

The idea is to give a *simple* framework to test function calls which
operate on buffers. This includes interactive function calls which may
request further input from the user. That is why I implemented the
advices to the reading functions.

> The function `ert-run-test-with-buffer' combines both features, and I
> don't think that's a good thing; it would be better to have a more
> primitive function `ert-run-with-interactive-inputs' (or similar) that
> doesn't do any of the buffer management, and let the programmer
> combine that function with `ert-with-test-buffer' and
> `ert-Buf-to-buffer' as appropriate.

It could be useful to separate this feature more clearly to make it
reusable in other situations. However, for testing interactive
functions on buffers - which to me seems a quite natural thing to do -
it should stay a feature of `ert-equal-buffer' /
`ert-equal-buffer-return'.

> Similarly, `ert-compare-test-with-buffer' looks like it checks a bunch
> of things that should probably be left as separate `should' forms on
> the caller's side.

Same as above.

> With functions like `ert-equal-buffer', your code introduces a notion
> of equality of buffers, and its definition seems somewhat arbitrary,
> so I'm not sure it's a good one.

True. It's just what I needed for my own tests. This is certainly an
aspect which deserves more work.

> For example, it doesn't take
> buffer-local variables or markers into account.

Yes. I already thought about this. As mentioned above one design goal
of ert-buffer.el was simplicity for the user. Thus the user needs a
simple syntax to write a buffer contents for a test. I used ?\^@ and
?\^? as simple syntax for point and mark which at the same time should
not collide with real input.

What could an extension to this syntax including markers and text
properties look like? For text properties I thought of something like
?\^[ and ?\^] as delimiters and some content describing the properties
between the delimiters. May be plain lisp forms?

> It should be easy to
> avoid the question of what the right notion of buffer equality is by
> letting the programmer extract ert-Buf data structures from buffers
> and compare those in `should' forms with some equality predicate.

I agree that the programmer should have a chance to define her own
equality but there should be a reasonable default.

> From a high-level perspective, testing point
> and mark looks like a small feature on top of testing buffer content,
> so I'm not sure it justifies as much additional machinery as your code
> seems to add;

Not if you test a function like `insert' which has a clear contract on
how point and mark is treated. I guess this applies to the majority of
functions which operate on buffers.

> we should look for ways to simplify things.

Indeed. However, my notion of simplicity seems to differ from yours.

> As a first step, could make the two features (providing interactive
> input and handling buffer content) orthogonal and send separate
> patches, perhaps simplifying the buffer content code after looking at
> how it's done in ert-x-tests.el?

I will give it a try.

>> The next step I considered was to support testing font locking - or
>> may be text properties in general. However, I didn't start this yet.
>> It certainly would be useful.
> 
> ert-x.el does have features related to this, see
> `ert-propertized-string' and `ert-equal-including-properties'.  ERT's
> self-tests make use of them.

I'll look at it.


                                                Grüße

                                                Stefan

pgpNJ2TzjIg7Z.pgp
Description: PGP signature