Re: [Gnu-arch-users] GCC v. Arch address@hidden: Regressions on mainline]

[Erik de Castro Lopo]

On Tue, 22 Jun 2004 19:15:47 -0700 (PDT)
Tom Lord <address@hidden> wrote:

> I thought about a PQM-driven Aegis-like protected mainline but I don't
> think it works out unless you do it in a _fairly_ hairy way.

Until very recently I worked with Peter Miller the main author of Aegis.
I forwarded him a copy of this mail and he repiled with the following
which he OKed for forwarding to the list.

Peter Miller <address@hidden> wrote:

> > I thought about a PQM-driven Aegis-like protected mainline but I don't
> > think it works out unless you do it in a _fairly_ hairy way.
> 
> Please expand what you mean by "hairy".
> 
> I like to think that Aegis has a clean but versatile process, grounded
> in the realities of real-world software engineering.  I could be wrong. 
> 
> 1: How you build:
> 
> Aegis is designed to watch the results of the build and will not advance
> the change set unless it builds.  This solves the problem where a
> developer commits a change set, and then the project no longer builds
> for anybody.  There is a checked-out copy called the "baseline" which is
> built as part of the commit... and since it isn't done in the
> developer's private work area, this also finds many "but it builds OK
> for me" class problems.
> 
> Assuming you trust your build (if you don't, see my RMCH paper) the
> builds can be incremental.  And if you don't trust incremental builds,
> what makes you think a full build will do any better, USING EXACTLY THE
> SAME DATA?
> 
> Assuming incremental builds most of the time (you can ask for a full
> build) and you are doing a maintainer build not a full GCC boostrap (the
> last time I build GCC it was over an hour for the full bootstrap) this
> build-during-commit is not the bottle-neck it may first appear to be.
> 
> The other thing you can do with a pre-built baseline is use it to "seed"
> your private work area, reducing the length of time taken by the initial
> build.  This could be a view-path, or hard links or copies, etc, etc.
> 
> Of course, you can set the build command to "exit 0" and it takes next
> to no time.  Of course, it can't detect any classes of common problems
> that way.  And it breaks the implied "commits must build" policy.
> 
> 
> 2: How much you test:
> 
> Quoting Tom Lord:
> > GCC commits happen too fast (last I checked) to serialise them while
> > inserting tests between each one.
> 
> Aegis is also designed to (optionally) require that each change set be
> accompanied by a new test.  This new test is required to PASS, it isn't
> just decoration.  Things like GCC are a pleasure to write tests for,
> because the inputs and outputs are so easily controlled.
> 
> If a change set is a bug fix, the test is also required to FAIL against
> the baseline.  This ensures that the test actually reproduces the bug...
> test fails on old code, passes on proposed code.  You don't have to
> write anything extra for this.
> 
> There is also a third type of testing.  Aegis accumulates all of the
> tests which accompany change sets into a regression test suite. 
> Developers can run this test suite at any time.  You don't have to write
> anything extra for this.  Project administrators can require that change
> sets pass the entire regression test suite before being committed. 
> However, this can take time is is not the default for all change sets. 
> When I started writing Aegis in 1990, I had recently worked on a project
> with 3 CPU weeks (yes, weeks) of regression test suite.  For some
> projects, running the entire test suite for every commit is not
> practical.
> 
> However, Aegis offers a happy medium: by using the change set meta-data,
> it is possible to form a correlation between source files and tests,
> allowing Aegis to suggest tests in the regression test suite you may
> want to run based on the source files in your change set.  Not as fine
> grained as code coverage analysis, but usually very useful.
> 
> Note that "make test" is usually the entire regression test suite.  As
> mentioned in the thread by Tom Lord <address@hidden> this is likely to
> make the queue grow without bound on any project the size of GCC.
> 
> Aegis also gives you the ability to run the regression test suite
> against the baseline without a change set.  Due to Moore's Law, that 3
> CPU week test suite I mentioned would now run in about 40 hours.  With
> Aegis, you could have a cron job run the regression test suite every
> weekend.
> 
> By using focused testing as described here, Aegis attempts to balance
> the need to commit in a timely fashion with the need to relentlessly
> test everything.  Thus test-during-commit is not the bottle-neck it may
> first appear to be.
> 
> 
> 3: Code Reviews:
> 
> By default, Aegis requires code reviews of all change sets before they
> are committed (actually, as part of the commit process).  My biggest
> concern is performing about 66 code reviews per day; humans scale worse
> than computers.
> 
> However, to more closely simulate the "look mum, no net" experience, you
> can configure your Aegis project skip the code review step.
> 
> Somewhere in my filing cabinet are several papers which purport to show
> that code reviews are your biggest bug catcher in any process, and this
> certainly dovetails with my own experience.  So I prefer not to skip the
> code review step.  But Aegis will let you if you want.
> 
> Of course, the "to enough eyes all bugs are shallow" mantra has the
> entire developer population doing the code review, but to my mind this
> is too late in the process for a commercial product, particularly when
> the PHB is likely to pop into your cubicle with no notice with a
> prospective customer or venture capitalist in tow and say "demo, now". 
> OSS projects with anonymous CVS access are eerily similar.
> 
> This is my major gripe with BK - it solves the wrong problem incredibly
> well.  It allows crap into the repository, and then spreads the crap to
> everyone really, really efficiently.  Surely propagating working code a
> little slower more desirable?
> 
> But see below for more of Tom's wisdom on this subject.
> 
> 
> 4: Tagging:
> 
> With CVS, in order to recreate the source at a given point in time, you
> have to tag ALL source files.  This doesn't scale - a commit is O(n)
> where n is the number of files in the repository, not the number of
> files in the change set.
> 
> Aegis assigns every change set commit a unique version number (usually
> called a "configuration identifier" or similar in the text books).  This
> version string (usually less than a dozen characters) can be used at any
> future time to reproduce the source of that version.  And it scales - it
> doesn't need to tag every file.
> 
> Of course, if you want your commits to take longer-and-longer as your
> project matures and grows, you can optionally tag every file at every
> commit, but it's not the default.
> 
> 
> Quoting Tom Lord:
> > Imagine yourself wanting to enact the [test before committing]
> > policy.   You make a GCC tree.  You add in your changes.  You want to
> > commit.   Before you commit, you run the tests.  The tests all pass,
> > fine -- but they took so long to run that now your tree is
> > out-of-date.
> 
> THIS can be a problem.  For any sufficiently large/active project, for
> any CM system.
> 
> And, as mentioned in the thread, a probabilistic approach is needed. 
> Unit test your change set.  Run 20 or 100 likely-to-be-relevant tests
> (the greatest number of relevant tests which will take less than 20
> minutes to run).  It isn't perfect, and it can't be, not without
> changing the process: slowing down the rate of commits, or using
> integration branches, or using pessimistic locking rather then
> optimistic locking (or no locking), or... etc.
> 
> 
> Quoting Tom Lord:
> > Quoting Mark:
> > > It is not acceptable to completely ignore this kind of
> > > breakage indefinitely.  Two months is far too long.
> 
> >  "Two months".   Implying that there is a shorter period of time that
> > is _not_ "far too long".
> 
> Tom is right.  My tolerance for my projects having a broken baseline
> (even in the time between releases and code freezes) is far less than
> Mark's, and Aegis (like Arch) is fussier than CVS about many things, but
> you *can* make a commit that will break things with Aegis.  It's just
> harder to do (just less *likely*) with Aegis, is all, and none of us
> want to intentionally break things anyway.
> 
> Or is the road to hell paved with probabilistic testing?
> 
> 
> Quoting Stephen J. Turnbull:
> > Then they could (optionally) check that the sources build, the tests
> > were run, whatever.  Implementation will be hairy, I think, but
> > conceptually well within what tla's framework is already capable of,
> > no?
> 
> Look how big Aegis is for some idea.  Not so much hairy, more "the devil
> is in the details".  Lost of details.  And you still have to answer the
> "what is it you want to work?" question - Aegis wants the proposed new
> head of the mainline or branch (as opposed to your private work area) to
> build and test successfully before the commit is allowed to complete. 
> And as a first approximation, it wants your private work area to do the
> same.
> 
> > you can state the problem under the literal policy, can't you?  The
> > real question is "must you test on the HEAD that you are proposing to
> > commit to, or is a test on your freshly updated workspace good
> > enough?", right?
> 
> Aegis' workspace update mechanism is slightly different, too.  (For
> those who don't know: Aegis work areas only contain the source files you
> are modifying and the implications of such, everything else is
> viewpathed from the baseline.)  This makes the commit window a slightly
> different shape.  Aegis does not demand that no commits were completed
> between the start of (update + build + test) and the start of a commit,
> only that the files in the change set are not out-of-date since then. 
> Problems are expected to be caught by the integration build + test
> (probabilistic, again).
> 
> But Aegis could demand that.  Hmmm... would projects use it?


Erik
-- 
+-----------------------------------------------------------+
  Erik de Castro Lopo  address@hidden (Yes it's valid)
+-----------------------------------------------------------+
"This is like creating laws against blasphemy and then complaining that
unbelievers can't come up with any logical argument against the existence
of God"  -- www.infoanarchy.org on the Digital Millenium Copyright Act