Re: testing framework and package.el

[Christian Ohler]

 On 2/10/10 13:53, Sebastian Rose wrote:
> 1.) Make testing a matter of one key-stroke
> To ease testing, I've written two little commands, as you can see on
> http://github.com/SebastianRose/org-test
>
>    `org-test-test-current-defun' and
>    `org-test-test-buffer-file'
>
> The idea is to search the directory structure from the current source
> file upwards for a directory named "tests/" if it exists.  Else ask the
> user.  Similar to what `add-change-log-entry' does.
>
> Below that directory, a tree like the source tree exists:
>
> project
>     +-- lisp/
>     |     +-- a.el
>     |     `-- b/
>     |         +-- b.el
>     |
>     `-- tests/
>           +-- a.el/
>           |     +-- tests.el
>           |     `-- a-defun.el
>           `-- b/
>               +-- b.el/
>                     +-- tests.el
>                     `-- b-defun.el
>
> If this setup exists, when editing defun-x in lisp/a.el,
> `M-x org-test-test-current-defun' will load tests/a.el/defun-x.el
> (fallback: tests.el there) and execute all tests with selector
> "^a-defun".
> `M-x org-test-test-buffer-file' in that same source file will load all
> *.el files in tests/a.el/ and execute all ERT tests for selector "^a".
>
> This would   a) recommend a certain structure for testing (what ever the
> structure will be) and   b) make testing a key-stroke.
This structure looks like it is optimized for many test files for each 
file of non-test code.  Right now, Emacs has 1735 .el files, and 35 
files total in the tests/ directory.  This ratio is not likely to invert 
any time soon.  The structure that you describe could work, but isn't it 
overkill?


> 2.)  Create temp buffers for output
>
> One thing I miss is a way to create output buffers automagically (I
> didn't look deeper into the sources yet).
>
> I know Lennart Borgman has written something for nXhtml.  It's called
> ert2.el and part of the nXhtml distribution (but I did not comprehend
> what's going on there yet - my elisp is badbadbad :).  Maybe Lennart
> will help here or even care for that addition?
Lennart's code is already integrated in ERT, obsoleting ert2.  I 
remember that I wasn't quite happy with his design, but didn't have time 
to fix it, and since it solved an important problem, I included it as an 
experimental addition in ert-exp.el.


> 3.)  Compare output to control files
>
> ediff the contents of those output buffers with control files.  We need
> this feature to test org's exports.  No one will find the time to sit
> there and search hundreds of output files for errors "by hand" (once
> tests exist).
What do you mean by "sitting there and searching hundreds of output 
files for errors"?  Do you have an example of tests where ERT's error 
reporting is too unspecific to be useful?

I remember writing some tests roughly like this:

(ert-deftest foo ()
  (with-temp-buffer
    (insert "initial buffer contents")
    (... invoke code under test ...)
    (should (equal (ert-filter-string (buffer-string) ...)
                   "expected final buffer contents"))))

Are you sure that you want the initial/expected buffer contents in their 
own files?  Such a split between the inputs and the invocation of the 
code under test seems cumbersome.  I guess if your inputs/expected 
outputs are complex enough, moving them out into files can make sense.  
Can you point me to some examples where you think the data is too 
complex to be in-line?

One limitations of my approach is that we really need some kind of 
"with-test-buffer" rather than "with-temp-buffer" that keeps the buffer 
around for inspection if the test fails, but this is item 2.) on your 
list already.  Another limitation is that the explainer for `equal' is 
not as pretty as ediff, but OTOH, as soon as you want to compare text 
properties, ediff can't help anymore (AFAIK), while there is an 
explainer for `ert-equal-including-properties'.

Text properties (e.g., font lock properties) are important to many kinds 
of tests.  Both ediff and using files are problematic in such cases 
because they don't support text properties.  Do text properties never 
come up in your context?

Christian.