Re: Item groups in bug tracker: "incorrect result" too broad, and masked by "regression"

[John W. Eaton]

On 01/11/2016 08:20 PM, LachlanA wrote:

> A discussion came up in a bug report in which A(x,y) is assigned a value,
> and after assigning to other elements of A,  A(x,y) has a totally new value.
> This is clearly a major problem, since it makes numerical results from
> Octave untrustworthy.
>
> This seems the sort of bug that should be flagged "incorrect result".  I'd
> like to find all such gross errors and try to fix them, but this is hard
> because "incorrect result" also sounds like the most appropriate item group
> for things like unexpected behaviour of the GUI.

Yeah, I didn't find better words for these things so mostly you see what 
I came up with.  I'd be happy for them to be better.  I don't expect 
that it will be possible to get people to choose the correct thing just 
because we have better words.  For example, I see that many people now 
think "crash" means "Octave gave me an error" instead of "Octave exited 
unexpectedly with a fatal signal".  But at least we could choose better 
categories for sorting once the reports are submitted.

> To make matters worse, the A(x,y) bug was a regression.  It is useful to
> know that, but flagging it as "Item group"="regression" hid the serious
> nature of the bug -- it was not longer "incorrect result".  Perhaps the
> seriousness could have  been flagged by the underused "severity" field, but
> that isn't something a submitter can't do.

Regression generally means a high priority bug to fix, but you may be 
right that it should be flagged in some other way, perhaps just by 
increasing the priority.

> I proposed splitting "incorrect result" into two: one (perhaps "wrong
> answer") clearly for numerical errors and one (perhaps "wrong behaviour")
> intended for errors in other settings.

But even if something is not a numerical function (GUI behavior, for 
example) incorrect behavior is still an incorrect result, I think.

> Michael Godfrey suggested that
> "wrong answer" may still be too broad and suggested "fatal error".  I
> personally think this sounds like either a crash of Octave or a  bug that
> crashes the users code, rather than a grossly wrong answer.  Perhaps "gross
> numerical error" is the right term, but to me that suggests something like
> amplified roundoff errors (e.g., 1 / (a-b) for  similar a and b).
>
> Does anyone have suggestions for this, or how to specify a regression
> without masking the other item-group values?  Does anyone have suggestions
> for how to prioritise debugging effort, given most bugs have the default
> priority and severity?

Other than that, I have no clue what is best.  Sorry.  But I did just 
notice that I recently added "Performance" to the list of Categories, 
when maybe it should be an "Item Group" instead?  I don't know.  I'm not 
even sure now what the difference between "Category" and "Item Group" is 
supposed to be.  Oh, I see that Savannah has the following descriptions 
on the page that is used to edit fields and field values:

  Category
    Select Box
    Generally high level modules or functionalities of the software 
(e.g. User interface, Configuration Manager, etc)

  Item Group
    Select Box
    Characterizes the nature of the item (e.g. Crash Error, 
Documentation Typo, Installation Problem, etc

Before choosing new names for things, or adding more names, maybe it 
would be best to define what it is you want to be able to categorize?

jwe