Re: [Gnu-arch-users] GCC v. Arch address@hidden: Regressions on mainline]

[Andrew Suffield]

On Wed, Jun 23, 2004 at 11:52:02AM -0700, Tom Lord wrote:
> 
>     > From: Andrew Suffield <address@hidden>
> 
> You idea goes in an interesting direction:
> 
>     > I'm thinking of a system along the lines of tinderbox, as used by
>     > mozilla - there is a group of hosts which just sit there endlessly
>     > building and running the test suite, against the most recent revision
>     > which has not yet been tested on this target platform (in practice it
>     > would be a little more intelligent than this, and allocate revisions
>     > to hosts so that test runs would tend to cluster on individual
>     > revisions, rather than being spread around).
> 
>     > When some revision reaches the threshold (for gcc this is "one clean
>     > test run on any hosted target"), it's merged. If any test fails
>     > (unexpectedly), then (a) immediate mail notifications are sent, and
>     > (b) a search begins, running just that test (and therefore faster),
>     > looking for a revision that either predates or postdates the
>     > failure. If it finds one, it runs the full test suite; if that passes,
>     > it's found an acceptable merge point. (This process runs in parallel
>     > to the endless builds of recent changesets)
> 
>     > This system has an interesting property: the queue never grows. It is
>     > held at an average of (commit rate) * (time for 1.5 build+test runs on
>     > the slowest required target) revisions behind everything else. That's
>     > going to be an hour or two for gcc - which is probably acceptable. By
>     > simply adding more hardware the system will fill in details for more
>     > revisions. Slower targets will fill in less revisions, but you still
>     > get consistent monitoring (and if you're patient enough, you get told
>     > which changeset caused the regression).
> 
>     > I've left out a whole host of details, 
> 
> Yes.  What I'm specifically not quite clear on is the branching model
> you have in mind.    I gather it's:
> 
> 
>       developer   via naked   dog pile   pqm +
>         branches   -----------> branch    ----------> monomain
>         (many)       pqm        (one)      testing
> 
> where "monomain" is a mainline that always lags a few hours behind
> dog-pile mainline but in exchange, for some definition of "quality",
> is of monotonically increasing quality.

Well, that's certainly one way to do it. In general there is no reason
why these can't be many-to-one combinations (I can't draw that in
ascii art, use your imagination), where several branches feed into
one. I'd say that the dogpile was just another "developer" branch,
though - the thing in the middle is the "patch queue" (that's where
the name originally came from, after all), and there's no reason why
anybody should notice it exists.

In the trivial case you just have one dogpile that feeds into the test
sequence - that's the clone of gcc's current scheme (just the right
half of your diagram).

At present I'm not actually worrying about that; you can construct
essentially arbitrary branch structures with these tools. I'm mostly
concentrating on the tools. Branching is your problem :P

> That would certainly be an improvement for volunteer testsers, 
> at least.

I'm thinking that it's of primary interest to release engineers, and
automated tools. You also generate a great deal of interesting data
through the process of constructing this branch - cross-reference bug
reports with test cases, and you've got a list of known bugs in any
given revision, and a list of revisions in which a given bug is
present.

Once it has become established, it can also become the focal point of
merging - you merge from the verified branch, not the dogpile, so you
get up to date with "the latest thing that doesn't actually
break". That's what you really wanted to merge anyway; if you were
after a specific feature then you'd have picked it from the feature
branch.

In theory you should still be able to commit directly into the
dogpile, but I haven't really tried thinking about that part yet. It's
not strictly necessary.

>     > but I think this scheme can
>     > work. Here's the last trick: it's completely orthogonal. "Proof of
>     > concept" takes the form of the system using the dogpile branch as its
>     > solitary feed; as developers observe that it works and works better,
>     > they can individually elect to work on a different branch and have
>     > that branch monitored for merging.
> 
> One thing I'm not clear about re GCC.
> 
> They _do_ do nightly testing.   They _could_ conceivably test more
> frequently.   Is the reason they don't the hw cost?  not wanting
> results any more frequently than that?   Difficulty in making the test
> runs less slow?   Tradition?  What?

Well, the approach I sketched out above relies on having useful
changeset boundaries. I don't think it could work with cvs (ignoring
cscvs).

It's *easy* to run a nightly cronjob. It's not easy to run tinderbox
(nasty piece of software), and I've never seen anything else that does
it. Also tinderbox is a good example of how *not* to implement it
sanely, even if it does follow the same basic principles - their
cvs-based system leaves them doing all the work by hand. In place of
tools, they have people continually monitoring and following policies.

Mozilla have demonstrated that it's certainly feasible for projects on
this scale.

I'm not aware if there's any reason why gcc in particular can't do
something like this, though. I tend to regard it more as a case study
than as a specific design target.

-- 
  .''`.  ** Debian GNU/Linux ** | Andrew Suffield
 : :' :  http://www.debian.org/ |
 `. `'                          |
   `-             -><-          |

signature.asc
Description: Digital signature