[Octave-bug-tracker] [bug #63848] Mean: some single class inputs can hit precision limit with sum

[Nicholas Jankowski]

Update of bug #63848 (project octave):

                 Summary: Mean: single class inputs can overflow with sum
producing wrong mean  => Mean: some single class inputs can hit precision
limit with sum

    _______________________________________________________

Follow-up Comment #8:

Hmmm, thanks. Didn’t think hard enough on what was actually am happening.
The output being ~half the true mean made me think it was the same as with,
e.g, (intmax+intmax)/2 overflow, where the fix had to occur in mean, not sum,
because the summation overflow was expected and the mean operation  ordering
just needed to be fixed (or type changed where possible). As I read bug #61143
Dmitri linked I see it is the small addend size running into the precision
limit of the growing summation. Retitling report. 

I assume we can hit the same issues for “double” through judicious choice
of inputs?

@dmitri, for the most part octave/Matlab does default to double output by
except in a few cases (and there is an outtype option to control this) so
trying to match expected output type is a compatibility issue that we
currently meet and I wouldn’t want to discard.


    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?63848>

_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/