Re: call for more ert tests

[Stefan Merten]

Hi all!

Being a relatively new Emacs developer my impressions might be
helpful for this debate. I'm subscribed to all(?) lists but usually
check only the subject lines for things interesting me. Also I'm
maintaining rst.el and so far did not do anything else in Emacs.

2 days ago Glenn Morris wrote:
> There are a lot of bugs in Emacs. [1]
> To me, too many of these feel like breakage of things that used to work
> before, because of implementation changes, or whatever.

I share this impression. And I think it's a shame that this is so.

> One thing that could help reduce this is more unit tests.

Definitely. Together with a build bot running tests automatically.

> If you haven't used it, ERT makes it pretty easy to write tests.

Some time ago I started to care about rst.el really. I wanted to do a
lot of changes in code I did not wrote myself and today this is
nothing to do without good test support. I found ert and it did most
of what I needed. It definitely is a very good foundation.

> Of course, many aspects of Emacs's behaviour are not easy to test

In rst.el I tested a lot of functionality which operated on buffers.
Thus I wanted to know that a buffer contained a certain content after
running a function. I wrote a little extension based on ert which
allowed such tests easily. I even suggested this code here but the ert
maintainer did not like it and suggested a different approach I never
checked out.

I think this can be done in a more generalized way. For instance the
next thing I was thinking of was to check for correct fontification -
since this is one of the major features in rst.el. I already thought
about ways of extending ert for this.

IMHO ert is a perfect foundation but needs extensions to test standard
cases where you are not (only) interested in the return values of a
function call.

> (GUI
> stuff, etc.), but many are. See test/automated/ for examples. [2]

Indeed they are. Often when I browse through the documented API I
wonder why there are no unit tests for it.

> I know writing tests is maybe not as interesting as writing shiny new
> features, but I think it will save work in the long run.

Well, I love writing tests. May be there are more like me?

Yesterday Stephen J Turnbull wrote:
> Does Emacs really need that high quality?

Definitely. Emacs is probably the oldest Free Software project and
IMHO it's a shame that the traffic on the development list is that
high because of bugs. IMHO a good part of this could be prevented by
suitable unit tests. Interactive testing is nice but definitely not
enough.

> That said, I encourage the core developers to lead by example,
> creating tests, even trivial ones, whenever possible.

One obvious task would be to have unit tests for the API specified in
the info documentation. There are also good chances that the
documentation can be improved by this.

> Today's general
> programming culture is very sympathetic to testing, even to the point
> of "write tests first".

Definitely. Frankly speaking I was shocked when I entered the Emacs
community and saw that in the end there are (close to) *no* tests.

> Testing is inherently boring
> work, exploring the things that shouldn't happen, and the burden is
> heavy, because writing good tests requires good specs, which we rarely
> have.  So designing those specs (and maybe even writing them down) is
> a hidden burden of testing.

I disagree that testing is boring. To me it's interesting to catch all
those corner cases in tests and improve the specification or even the
implementation as a side effect. May be this is a tester approach
(which I'm not ;-) ).

Yesterday Lars Magne Ingebrigtsen wrote:
> ert is fine, but, I think, somewhat misguided.  It allows us to test the
> functions we have Lisp interfaces for, but not the deep internal C
> bits.  And that's kinda backward.

May be I do not understand Lars' intentions here, but what is testing
the C code good for at all? I mean there is an API - or rather: lots
of APIs - and *this* is what needs testing. The C code to me looks
like an implementation detail I'd not write tests for. Or at least
separate those tests clearly from the API tests.


                                                Grüße

                                                Stefan

pgpThzCl8gXMX.pgp
Description: PGP signature