Re: stable vs. experimental archive

[Jaroslav Hajek]

On Wed, Apr 22, 2009 at 5:44 PM, John W. Eaton <address@hidden> wrote:
> On 22-Apr-2009, Jaroslav Hajek wrote:
>
> | Given the apparently somewhat less active developer base of Octave, I
> | think we can aim for something a bit simpler, and maybe more
> | collaborative.
> | Let's just have "main" and "stable", with all active developers having
> | access to both "main" and "stable".
> |
> | When someone fixes a bug, he pushes the fix to "stable". When he/she
> | makes a new feature, improvement or whatever, he pushes it to "main".
> | If it's highly experimental, he'll develop it in a separate repo, and
> | then it will be merged (for big projects) or rebased in main.
> | "Main" will often pull from "stable" and merge, to get the bugfixes.
> | Anyone is allowed to do it any time.
> | Once per time, stable will pull from "main" and merge, to get new
> | features once they become somewhat tested and agreed upon. This should
> | happen only after discussion.
> |
> | >  Bugfixes are pushed to -stable, then pulled and merged to
> | >  mainline. No new features are going to stable, mainline is pushed to
> | >  stable after each release.
> |
> | OK, that matches my model above.
>
> I'm not sure how well this will work in practice for us.  I posted it
> as a possibility, without really thinking about it too much.  Some
> problems I see with it are things like
>
>  A major change in the main branch (like the symbol table changes I
>  applied after 3.0, or the recent switch to a tree-walker
>  interpreter) will mean that a bug in those parts of Octave will need
>  to be fixed in completely different ways in the two branches.  So
>  pulling and merging will not be trivial in those cases.
>

This is unavoidable in general - when such a change comes into play,
somehow somewhere someone needs to prepare two different fixes for the
same problem, there's no way around that. In my model that would
result in a conflict.

>  Without some discipline in what is added to the main branch, we will
>  still be in a state where what is in the main archive can't be
>  released.

I thought it's the stable branch that's supposed to be released.

>
> I think I would prefer the following structure, which is mostly like
> what we have now:
>
>  A set of experimental archives that can be used for doing whatever.
>  I don't care who hosts these, but we can link to them on a
>  developers page on the Octave web site.  Although links for these
>  may appear on the Octave web site, these archives are not "official"
>  Octave project archives.
>
>  A main development archive.  Releases are made from this archive.
>  Periodically, we have feature freezes, which should not last too
>  long, but are necessary to stabilize for release.  Changes to this
>  archive should only be patches that are relatively safe, or have
>  been fairly well tested.  We should try to keep this in a state
>  such that releases are possible most any time, or at least with only
>  a short feature freeze period.
>

Which brings us back to the main problem I wanted to solve: that the
feature freeze may interfere with someone's time for active
development. But OK, let's see how the experimental archives solve the
problem.

>  A stable branch for the current stable release.  The only changes
>  that should be applied are for regressions or documentation errors.
>  These branches may be kept as internal branches of the main archive
>  instead of being separate archives (if we choose that, then I guess
>  I would like to reattach your 3.0.x branch into the main archive on
>  savannah).

The main problem I had with the current scenario is that it seemed to
me people mostly ignored (or were unaware of) the release-3-0-x
archive at Thomas Weber's site. If more people built from it, there
would have been far better chance to catch bugs introduced by bugfixes
(such as the CR/LF issue).

> The main change here is elevating the awareness of other experimental
> archives, and using them for the real development work.  It's up to
> the people working with those archives to synchronize with main, and
> to export patches for the main archive that will apply cleanly.

XXXXXXXXXXXXXXXXXXXXX

What I mainly want is that the parallel development doesn't introduce
unnecessary extra work.
Suppose I start developing some improvements in my parallel repo, that
will last for a month.
During that month, whenever bugfixes arrive in the main archive, I
want to easily get them.
What exactly am I supposed to do? Say I have already committed 5 new
patches in my archive, and then 2 bugfixes arrive in the main branch.

1. The easiest way is to do a pull and merge. If the changes are
orthogonal, the merge will be a no-op.

2. Another possibility is to keep my 5 patches in mq, and when new
changes arrive from the main branch, just qpop them, pull, update,
qpush them back. If the changes are perfectly orthogonal, then the
pushing may be also a sequence of no-ops, especially if with my patch
for Mercurial that does context-free diffs of prepended ChangeLog
entries. But if they aren't, I may end up fixing conflict 5 times in
the same piece of code, because each of those 5 patches did something
there. So, in fact, I should best avoid having more patches dealing
with the same part of code. In the end, I may spend more time
designing patches than doing the actual programming, and that's bad.
Another big drawback is that mq are stuff for local development -
they're not visible in any repo so that people can pull them.

I don't basically see other reasonable options, and it seems to me
that 1. is a big winner.

XXXXXXXXXXXXXXXXXXXX

> I know this will be slightly more work than just pulling from other
> archives, but I would really prefer keeping the main archive history
> relatively uncluttered, primarily because then it is easier for me to
> see that the changes that are going in are ones that we actually want,
> that they don't include some extra experimental baggage, and that they
> don't have unintended consequences (like wiping out someone's previous
> changes).  If I just pull from some other archive, how am I to know
> that I'm not picking up some extra garbage along with the changes I
> want?
>

You can simply ask, can't you? But in my model, if you pulled from
main to stable, people working with main should together have a very
good idea what has been changed. In my model, main shouldn't be for
everyone to try every single idea, but to develop features that have
been agreed upon and should eventually go into Octave.

> If you think that requiring a relatively clean history is a bad idea,
> can you explain why having a cluttered history in the main archive is
> a good thing?  I just don't see it.  Faster, wilder development in the
> primary public archive is not necessarily a good thing.

I'm not sure what you mean by cluttered history. If I make pull &
merges in my experimental repo, then the history will inevitably
contain merge changesets. The reasoning why pull & merges are best for
longer parallel development are given above. So, let's carry on the
mental experiment: we arrive at the point when my changes should go
into the main archive. What am I supposed to do?
Note that now there is a branch (later merged) for each of the pulls
from main repo I did over the month, so there are, say, 10 branches.
Rebasing them will be a real pain, especially if conflicts occur.

What I proposed was actually a compromise - we allow pulling from main
to stable. The main archive is public and controlled, so you *do* have
some control what features are there.

When I start development of a new feature *with my model*, I will do
it in small chunks and always rebase them when pushing to the main
archive. The main archive is unaffected by feature freeze, so I may do
it as often as I want, thus I can avoid the rebasing becoming too
painful. This is actually what I do now with the main archive, but has
the negative effect that the main archive is not stable enough.

This means that most of the development will happen in the main
archive, which is IMHO a relatively good thing, because people have a
centralized place to look for new, experimental features rather than a
dozen of individuals' repos. Only highly independent or experimental
efforts (like, say, developing a new interpreter) will be deferred to
external repositories.

So, if by "cluttered" you mean "not linear", then that is just a
necessary sacrifice, there's of course nothing good in it per se.

Does it make more sense now? If you still disagree with the reasoning,
then please explain particularly your view on the paragraph marked by
XXXXXXXXX above.


> | > If we have other archives that are used for experimental features,
> | > then I think we have a better chance of keeping the main archive in a
> | > state that can be released at almost any time.  But doing that
> | > requires some discipline, and not just pulling in random changes from
> | > experimental archives as soon as they have been committed.
> |
> | Yes. Every main -> stable pull must be announced and discussed.
> | stable -> main pulls are allowed any time.
>
> Do stable -> main pulls always make sense?  We tend to have situtions
> like the symbol table or interpreter changes that cause the two
> branches to diverge.  It seems to me that pulling would not be
> appropriate in those cases.
>

Again, see above. Creating two different patches in such case is
inevitable. In my model, this will also result in a conflict, but I
think it will be easily resolved. Maybe mercurial offers something to
help (hg revert, for instance). In any case the problem will be solved
in the main archive, stable archive will not care about it.

-- 
RNDr. Jaroslav Hajek
computing expert & GNU Octave developer
Aeronautical Research and Test Institute (VZLU)
Prague, Czech Republic
url: www.highegg.matfyz.cz