Re: [Qemu-devel] Suggestion for testing framework

[Ian Jackson]

Paul Brook writes ("Re: [Qemu-devel] Suggestion for testing framework"):
> We were talking about a tester that does periodic long running tests off svn 
> trunk, and reports the results. Individual developers are not directly 
> involved.

Just reporting the results is all very well, but as others have said
testing is only useful with a very firm commitment to deal with the
results of reports.  Few Free Software projects can manage to do this
without some kind of formal and automatic linkage of QA passes into
code distribution or propagation.  Without that, continuous discipline
is needed - and if it ever slips, the test failures become an
ever-deepening swamp.

One common approach to this problem used with success in very
different ways by both Debian and Xen and probably many others, is to
maintain parallel branches: one (call it `unstable') is changed
freely, and the other (call it `testing') is updated from unstable
only when the QA criteria (whatever those are) are met.

If there is some reason why the testing version has to be up to date,
this naturally leads to developers working on fixing test failures
because they end up blocking on it.  In Debian this is achieved (more
or less) by the fact that `testing' will form the basis of the next
release and is the only route to wide deployment; in Xen the testing
tree (called `xen-unstable') is actually the one that's widely
publicised and against which all submitters are expected to submit
patches.

I'm not sure exactly what would be the best corresponding model for
Qemu.  One idea might be to decree that while the testing tree is
broken, changes other than ones to fix it are forbidden.

There is one other unfortunate problem with the scheme above: this
mode of working is much harder to support without a version control
system that does merge-tracking.  With such a vcs, everything can be
largely automatic, including adjusting patches submitted against
testing to apply to unstable.  With svn such a workflow would be
nearly impossible.

Ian.