[Orgmode] Re: Testing --- again...

[Sebastian Rose]

Carsten Dominik <address@hidden> writes:
> Hi Sebastian,
>
> the lack of a testing suite for Org-mode is really frustrating,
> and even more frustrating is that we have had like seven attempts
> to start one, and each of these lead to nothing.  So I would
> be perfectly happy to give a free hand, write access to the repo
> and a full directory in the distribution to implement one.
> Once there is a framework, I am sure many people would be
> willing to contribute tests.


There was no support for testing in Emacs and I could not see something
coming up.  I guess that was one of the main reasons, that nothing
happened.  The existing tests use a undocumented framework wich consists
of many source files itself, so that I was not able to get my head
around that in half an hour (which is by far too long for most of us).

I asked on emacs-devel and Stefan Monier told me that there was some
consensus that ERT _should_ go into Emacs (24?) or at least to
elpa.gnu.org.  Christian Ohlert already sprang into action, so I'm
optimistic that ERT is the right choice on the long run.


So.  The framework will be ERT.  I'll just right a few commands and
functions that make it _dead simple to right and execute tests_.


>
> More comments below.
>
> On Oct 2, 2010, at 5:51 AM, Sebastian Rose wrote:
>
>> Hi,
>>
>>
>> I thought about testing again recently.  This is something, that never
>> really got started.  For a reason:  there's no framework for testing.
>>
>> I therefore wrote a very rough proposal,  found  on
>> http://github.com/SebastianRose/org-test
>>
>> The idea is, to provide two simple commands:
>>
>>
>>  *  org-test-test-current-defun
>>     will search for tests for the defun point is in or behind
>>     (`beginning-of-defun') and execute them surrounded by
>>
>>  (let ((select (or selector "^org"))
>>      (deactivate-mark nil))
>>    (save-excursion
>>      (save-match-data
>>
>>
>>  *  org-test-test-buffer-file
>>     will search for tests for the entire file and execute them the same
>>     way.
>
> FIrst:  I have *no* clue about testing.
>
> Second, I am surprised that you want to structure it by function.  I would 
> have
> thought that it could be structure by file at the most.  And then there will
> be tests that involve code from many files.


It's both actually (see below).  But a simple function could easily have
5 or more invariants.  For each invariant a test should be written.
That makes 5 tests.  Having those (_optional_) per function test files
will help finding certain test easily.  Some functions will also need
lot's of other test code, e.g.  let bindings that emulate different
setups or code that creates temporary directory structures (publishing)
and the like.

Naming (again: optionally) files like the functions they are written for
and putting them into a directory named after the code file makes things
easy to find --- even without docs.


>> ...

>> The idea is to search the directory structure from the current source
>> file upwards for a directory named "tests/" if it exists.  Else ask the
>> user.  Similar to what `add-change-log-entry' does.
>>
>> Below that directory, a tree like the source tree exists:
>>
>> project
>>   +-- lisp/
>>   |     +-- a.el
>>   |     `-- b/
>>   |         +-- b.el
>>   |
>>   `-- tests/
>>         +-- a.el/
>>         |     +-- tests.el
>>         |     `-- a-defun.el
>>         `-- b/
>>             +-- b.el/
>>                   +-- tests.el
>>                   `-- b-defun.el
>>
>> If this setup exists, when editing defun-x in lisp/a.el,
>> `M-x org-test-test-current-defun' will load tests/a.el/defun-x.el
>> (fallback: tests.el there) and execute all tests with selector
>> "^a-defun".
>
> Well, OK, this is fine.  But under a.el and b.el there should also be
> general tests that are not function dependent, and there should be a place
> to put tests that you do not want to assign to a specific file.


That's what tests.el is for currently (besides that it is the fallback
for function tests).

`org-test-test-buffer-file' simply loads _all_ *.el files in a file's
test directory.  With prefix argument, it loads tests.el only.



> We do have a "testing" directory already, you can use that.
> I would prefer the tests to be in testing, not in lisp/testing
> if possible. I would like to have the lisp directory contain
> only code.  If possible.


That's already possible.  The directory "tests" (will rename that to
"testing") is searched in the same directory as `buffer-file-name' and
from there up the directory tree.  It could be place entirely outside
the project tree.

> It would be OK to have a lisp subdirectory in testing,
> just as it would be OK to have contrib/lisp in testing
> for the contributed packages.

That's what it has to be like (it is).  We couldn't use the "autmatic"
test search otherwise.  E.g. emacs/lisp/ has many sub-directories each
of which could hold files with the same basename (in theory).


The above graph was wrong, sorry.  This is the current structure:

...

 `-- tests/
       +-- lisp/
             +-- a.el/
             |     +-- tests.el
             |     `-- a-defun.el
...



It's supposed to work for many projects, too. See how the directory
trees are rebuild below "testing/":

projects
  +-- emacs/
  |     `-- lisp/
  |          `-- org/
  +-- some-project/
  |         `-- code/
  |               `-- a.el
  |
  +-- testing
        +-- org-test.el
        `-- tests/
              +-- emacs/
              |     `-- lisp/
              |          `-- org/
              `-- some-project/
                        `-- code/
                              `-- a.el/
                                    +-- a-defun.el
                                    `-- tests.el

This kind of structure is the only way to prevent file name conflicts
and still act completely automatic.

It will _not_ be possible to have testing/  in a subdirectory
org-mode/lisp/.  But who would want that?   ;)


> PLEASE, go ahead.  I do not think you have write access
> yet on repo - give me your user name and I'll activate you.
>
> - Carsten

I'd be glad to care for the testing.  Albeit I'm not sure if I will do a
"good" job there.  But I see the urge to get that started, and that I
can do.  And, as always, I would enjoy helping where possible ;)

This would involve to scan the bug reports for regressions and right
tests accordingly.

My user name on repo.or.cz  is "SebastianRose".



Here a short (short is good) list of what the "testing" directory should
contain (please edit):

  - A subdirectory "tests" where all the tests go.
  - Some concise docs to get hackers started with testing.
  - The file org-test.el (since it's only needed if "testing/" exists).


What we should do, is to strip the org-mode/testing/ directory from
release packages.

Ahhh! "makerelease" already cares for this :)




  Sebastian