[Octave-bug-tracker] [bug #54567] median(uint8) gives bad results

[Dan Sebald]

Follow-up Comment #12, bug #54567 (project octave):

BTW, we should at BIST tests for all these cases.

Also, we know mean() probably has the issue we are trying to avoid:


octave:253> mean([intmin('int64')+5 intmax('int64')-5],'native')
ans = 0
octave:254> mean([int64(-1) int64(0)],'native')
ans = -1


I.e., the large magnitude int64 numbers have a loss of precision in mean() so
the result doesn't match the formula of Comment #11.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?54567>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/