Re: GOP2: 2 - Stable releases and roadmap (radical change)

[Janek Warchoł]

Hi All, Graham,

first, let me apologise for not responding promptly.
Secondly, here's my reply to Graham's almost-original proposition;
i'll send a reply to current discussion ("Clear policy discussions")
separately.

On Tue, Jun 26, 2012 at 10:55 PM, Graham Percival
<address@hidden> wrote:
> Let’s drop the “any unintended change” thing, and go totally with
> the regression tests. Tests pass? We can make a stable release.
> Also, let’s have an official roadmap.

In general, sounds reasonable but not perfect.  See below.

> *** Motivation
>
> There seems to be widespread frustration with the current system.
> At the moment, any “unintended change” blocks a release (plus a
> few extra conditions), so we’re at the mercy of all sorts of
> behaviour that isn’t covered by the regtests. This makes it hard
> to plan ahead for everybody: developers wanting to work on large
> features or refactoring, users, linux distribution packagers, etc.

Yup.  The problem with current system is the "surprise" part: we know
things break, but it's annoying when they disrupt our plans.

> *** Details: Regtests
>
> The current regtests don’t cover enough – that’s why we keep on
> finding new regression-Critical issues. I think it’s worth
> expanding the regtests and splitting them into multiple
> categories.
> [several types described in http://lilypond.org/~graham/gop/gop_3.html]

This is a really good idea, Graham!  +10

> In cases where the output may look bad because it is a stress test
> (e.g., ‘break.ly’, ‘spacing-strict-spacing-grace.ly’), this fact will be noted
> in either the texidoc or as a markup inside the score.

i vote for markup inside the score.  Much more convenient imo.

> *** Programming regtests
>
> To avoid slowing down programming to a crawl, I figure that we’ll
> identify some subset of these regtests and have a separate make
> regtests-quick command which only evaluates that subset.
>
> As a rule of thumb, I’d say that the regtests-quick target should
> take as long as a make from scratch.

*very* good idea!  +20

> git master should always pass all regtests. If it
> doesn’t, then reverting should be the first option.

+1

> *** Roadmap
> The 3.x series will consist of a series of random breakage from
> functionality not covered under the existing regtests and from
> manual .ly changes required by GLISS.
> [..]
> We’ll then have another few months of GLISS discussions, then a
> pause for implementions, then 3.4. Repeat as necessary.

That's ok with me.


On Tue, Jul 10, 2012 at 11:50 PM, Trevor Daniels <address@hidden> wrote:
> So far there have been c. 75 critical regressions under the
> current definition of 'critical' since 2.14.  All but one have been
> fixed, many of them promptly.  This prompt attention IMO
> is due only to the fact that they were deemed to block a
> stable release.  If the only criterion is that the release compiles
> the (extended) regtests satisfactorily, then I doubt that adequate
> attention will be directed to bugs discovered after the release
> that would be deemed critical on the current definition.  That
> would seriously degrade the quality of our stable releases.

This is a valid concern.
What about something like this:
when a regression against latest stable is found, it's not marked as
critical (as Graham suggests).  However, when we make a stable
release, all regressions present in the tracker become critical.  In
other words, if current unstable is, say, 2.17.x, regressions against
2.16 aren't critical (don't prevent releasing 2.18), but still-unfixed
regressions against 2.14 are critical.
This way we are "forced" to fix all regressions (sooner or later), but
we eliminate the element of surprise that is so annoying in current
system.  Things may break, but we won't leave them broken too long.
Since the lack of "surprises" should allow frequent stable releases,
all regressions should be fixed pretty quickly.  We could actually
adopt a policy of aiming to have a stable release each 3 months, which
should help with that.

How do you like it?
Janek