Re: policy for release branch

[Robert T. Short]

I think some of what I am going to say just mirrors John's comments, but 
I have a couple
of other things to think on as well.  My opinions are just that, 
opinions.  Feel free to
ignore any or all of them.

Jaroslav Hajek wrote:
> hi,
>
> while I realize it's mainly up to me (since I'm doing the job), I want
> to initiate a discussion about the topic.
>
> When a release is branched, what patches can go into the release
> branch? Unoficially, it should be bugfixes and docfixes, but this
> specification is still vague. Is missing Matlab
> functionality/compatibility a bug?
>
>   
I think the only things that should be fixed are serious problems.  
Missing features
or compatibility issues don't count.  Even minor and obscure problems 
should probably
be left alone.  If we had a QA staff and the resources to handle 
multiple branches it
would be different, but we don't.

Of course somebody has to define what "serious" means, but I would tend 
to err
on the side of making it easier on yourself.  I am not sure that even 
crashes should
be fixed, but wrong answers probably should be.  It all ends up being a 
judgment
call and yours is as good as any.

Introducing incompatibilities is a big downer.  Unless the bug is 
horrendous don't
create incompatibilities.  If an incompatibility is introduced at a 
major release,
that is the price of open-source software, but minor releases should 
stay reasonably
stable.

Another thing you might think about is soliciting help to manage the 
release.  I would
certainly be willing - like everyone else my time is very limited, but I 
could spend
some time.

> I think there was also a proposal to only fix regressions; but this
> doesn't work well when new & buggy features come out with a major
> release - surely we don't want to leave them buggy until another major
> release.
>
> Another point is about not breaking the binary interface between minor
> releases; this has some merit w.r.t. packages compatibility; however,
> what if an important bugfix is created that breaks the interface?
> Should it be left out? Or adapted? Who will do it?
> Speaking about this, how does one tell whether the change breaks the
> binary compatibility? Modifying methods is sure; what about adding new
> ones? What about changing classes that are not marked as API - is that
> OK?
>   

> Wow, what a lot of questions. Before attempting to answer some, here's
> a major thesis: *OCTAVE CAN'T COMPETE WITH MATLAB IN TERMS OF
> STABILITY*. I really mean it - look how every major release introduces
> a plethora of new bugs; sometimes new bugs pop up even in minor
> releases. It's not surprising - Mathworks probably has paid testers,
> long testing cycle and much stricter development rules and likely a
> large test suite. Surely Matlab has also bugs, but they don't get into
> a release that easily. So, Octave just can't compete with that.
>
>   
I think a more general statement is:  octave can't compete with MATLAB. 

What octave provides is a good, solid tool that does a lot of what MATLAB
does but at a substantially lower cost.  Feature requests and bug reports
to The Mathworks seem to go into the ether.  On the other hand, the
octave community is actually interested in what the user community
has to say, and if a user wants something, well, all she has to do is lend
a hand.

One implication of this is that we need to be very choosy about how hard to
chase MATLAB.  I think the octave community has done a very good job of
striking a workable balance.  That balance is achieved primarily because 
folks
that want something get in and do it.  That way the important stuff gets 
done
and the less important stuff doesn't.

What we CAN do is balance between features and stability.  The harder we 
push
new features, the worse the stability problem.  If we fuss too much 
about stability,
then we will never get new stuff done.

I think recent octave has pushed features over stability a little too 
much.  That is
my *opinion*, not a fact.

I really think we need a more measured release process.  We froze the 
development
for all of two weeks or less and called it a release.  We should view a 
release as
a chance to take a deep breath and really look for bugs - everybody gets 
involved.
With an all-volunteer development crew, you can't MAKE people do anything,
but I think almost everybody would be willing to spend some time running 
their
favorite scripts and test cases and making sure things seem to work.  
Just a
simple and basic QA process would help a lot, I think.

If we slow down the release process, then bug fix releases will happen 
less often,
but should be less buggy and that seems like a good trade to me.

Just my opinion. 

> Where it *can* compete, is that most reported bugs are fixed within a
> week, often just a day or two. Now this is, on the contrary, something
> Matlab can't compete with - you can get a fix very fast, or even an
> improvement or a new feature. This is, however, only true for that
> relatively small number of users compiling Octave from sources. Those
> who use, say, packages from a Linux distro, are limited by releases of
> Octave and package updates in the distro. It would be surely nice if
> those users experienced, at least partially, the advantages of the
> fast bugfixing/development pace of Octave. The problem is that this
> clashes with some of the above policies: if a fix breaks binary
> compatibility, you'll need to wait for a major release to get it.
> Maybe you don't care about the binary compatibility, and you're fine
> with reinstalling packages; still, you have to wait. Is that an
> acceptable price for such a policy?
>
> I guess it all comes down to the question how often the major/minor
> releases are. Anyone can probably wait 6 weeks for an important bug
> fix; will anyone wait 6 months?
>
>   
So, we are now back to the question "how often".  Important question.  I am
not in favor of 6 week releases since I would rather see each release get a
little QA time.   Furthermore, there is some effort involved in packaging
octave for a distribution and too-frequent releases will tend to overload
the process.  Two to three months seems ok to me.

The other serious issue is maintaining two source trees.  How often should
we make a major release?  That is, a release with new features that might
break some existing code.  Certainly when there are new features that really
enhance the octave experience.  Probably when the pain of managing
two sets of patches gets to be more trouble than it is worth.

Just some thoughts!


Bob