Re: ert.el --- Emacs Lisp Regression Testing

[Christian Ohler]

Richard Stallman, 2007-12-31:

> ert.el looks very useful.  Would you like to contribute it to Emacs?

Yes.


> One of the things that has held us back from setting up a test suite
> in the past is that only a minority of Emacs bugs concern Lisp progrmming.
> Most Emacs bugs involve keyboard commands and/or redisplay.

This may be true of the core of Emacs, but testing is also important for 
(third-party) Elisp packages, and they tend to be testable from Elisp.

Also, automated testing is not only about bugs.  A comprehensive test 
suite gives you the freedom to modify any part of the code without being 
afraid that you'll break something without noticing.  This makes it 
easier to add new features or improve the implementation of existing 
features and will be useful for Emacs, too, even if we only use it for 
things that can be tested from Elisp.

But, yes, we should also try to automate tests that involve input and 
redisplay.


> It occurs to me that M-x term could be used to test for redisplay.
> The idea is that you run a sub-Emacs, send it commands, and check what
> it displayed (because what it displayed is now in a buffer).

Does a significant share of the input/redisplay bugs that occur in a GUI 
Emacs also occur on the terminal?


> It would be terribly slow to do this starting a new sub-Emacs for each
> test.  So the test framework ought to be designed to do many such tests
> with the same sub-Emacs, clearing out whatever is necessary each time.

Yes.


> I think that an extension to ert.el, for sub-Emacs testing, might make
> it easy to define those tests, much as ert.el makes it easy to define
> tests of Lisp execution.
>
> Interested?

The first step would be to identify a few bug reports involving input 
and redisplay that should be reproducible under M-x term and should make 
good candidates for automation.  Then we can look into automating them 
and try to come up with a general design.  I don't think I'm the right 
person for that first step, though.

Christian.