Re: testing framework and package.el

[Christian Ohler]

 On 5/10/10 14:09, Stephen J. Turnbull wrote:
> Christian Ohler writes:
>
>   >    On 5/10/10 2:18, Chong Yidong wrote:
>   >  >  Christian Ohler<address@hidden>   writes:
>   >  >>  Is there a deep reason
>   >  >>  why Emacs currently has test/ as a separate top-level directory?
>   >  >  Because the test files are not to be distributed with the tarball.
>
>   >  Why not?
>
> Conceptually, tests should be developed separately from the core, just
> as the Lisp Reference is.  Of course, in practice it's not that easy,
> but I think it's a good idea to structure the tests and the reference
> at the same level as the core sources.
>
> XEmacs experience with its tests has shown that having the tests
> subsidiary to the source tree leads to regressions.  Specifically,
> changing the source code leads to changes in behavior, and these
> changes are "sanctified" by writing tests after the fact.  But these
> tests are *not* run against older versions, leading to unplanned
> divergence in behavior.  Occasionally these divergences are annoying
> to users or third-party libraries.

AIUI, XEmacs has its tests in a tests/ directory next to lisp/, in the 
same repository.  Are you saying this has proven to be a bad idea?

The most frequent use cases of the testing system are probably browsing 
existing tests, writing new tests, running tests against locally a 
modified Emacs, and maintaining tests.  We should lay things out in such 
a way that these actions are as easy as possible.  I expect that running 
the current tests against older Emacs versions is something that we will 
want to do much less frequently, so we should not optimize for this use 
case at the expense of the others.

Anyway, I don't think the directory layout matters much as long as there 
are very few tests.  I went with a lightweight solution and put the 
tests all in one directory test/automated/ like XEmacs does.  (I'm 
referring to the current state of the ERT repository here, this code 
isn't installed yet.)  I think the primary use cases would benefit from 
having the tests right next to the corresponding sources rather than in 
a separate tree, but I'm not going to fight over it.  (Emacs functions 
for jumping from a file in lisp/ to a corresponding file in test/ solve 
only part of the problem since they only help when viewing or editing 
the files, but not while operating on them with shell commands, VC, or 
other tools.)

We can always move the files around later if we decide on a different 
convention.

Christian.