Re: testing framework and package.el

[Christian Ohler]

 On 14/10/10 1:19, Stephen J. Turnbull wrote:
> Christian Ohler writes:
>
>   >  IIUC, this design requires developers who want to add features and write
>   >  tests for them to check out two different bzr branches and submit two
>   >  different patches.
>
> Currently, yes.  With nested branches, the checkout will be implicit.

Let's find a solution that works well now rather than speculating on a 
future bzr feature that may be delayed, may be slow in its initial 
implementation, or may turn out not to work the way we think it will.  
Once nested branches are implemented, we can learn how they work, and 
then split the repositories if it still makes sense.


>   >  Isn't that a rather high bar that will be inconvenient for
>   >  developers and discourage them from writing tests?
>
> In my experience, people who actually write tests (unless there's a
> firm requirement that tests be submitted before committing) are hard
> to discourage.

We want more than this; we want those who don't write tests to start 
doing so.  Putting significant barriers in their way makes it much less 
likely that Emacs will gain a useful test suite.


> The real problem is getting those tests backported.

You mean, running a test suite in an older Emacs version, and adding 
fboundp checks to the tests that test missing features, is harder than 
writing the test suite in the first place?  I don't expect that will be 
true.


>   >  As an alternative that avoids this problem, we could maintain the tests
>   >  in Emacs trunk, but our makefiles could have a test_src variable that
>   >  defaults to tests, and that we can set to ../trunk/tests when running
>   >  tests in the emacs-23 directory in Stephen's layout.
>
> Which still requires serious developers to have multiple checkouts.

You are right that developers who want to run tests in multiple versions 
of Emacs will need the code for all those versions, and the code for the 
tests.  There is no way around that.  However, including the tests in 
Emacs' trunk  saves them one checkout, and means less hassle submitting 
Emacs bugfixes together with additional tests.  Also, casual 
contributors often won't have multiple Emacs versions checked out, and 
they are worth catering for.  It seems more likely that they would 
contribute tests if they can do so without the overhead of multiple 
workspaces.

The solution of adding a makefile variable like test_src that allows 
running the test suite from a different workspace seems to solve the 
problem you described, and is much simpler than splitting the 
repository.  Do you think it has significant drawbacks?

Christian.