[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Octave-bug-tracker] [bug #54567] median(uint8) gives bad results
From: |
Kai Torben Ohlhus |
Subject: |
[Octave-bug-tracker] [bug #54567] median(uint8) gives bad results |
Date: |
Sat, 25 Aug 2018 08:12:20 -0400 (EDT) |
User-agent: |
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36 |
Follow-up Comment #5, bug #54567 (project octave):
Good catch with the (u)int64 data type.
I think the second version of comment #2 will deal with averaging two numbers
for the even case with (u)int8-32, single, double, and logical. For (u)int64,
we might have to introduce an exception
## Perform the averaging in double precision to avoid intermediate
## overflows of finite data types like "int8". For "int64" and
## "uint64" double precision is insufficient and we have to
## perform the division by two beforehand.
x = nth_element (x, k:k+1, dim);
if ((isa (x, "int64") || isa (x, "uint64"))
&& (any (x >= flintmax () / 2)))
warning ("median: inaccurate result due to large integer values.")
retval = sum (x / 2, dim);
else
retval = cast (sum (x, dim) / 2, class (x));
end
The formula of comment #3 and comment #4 is hard to implement regarding
multi-dimensional arrays. If you found a more elegant way, I would be very
happy ;-)
_______________________________________________________
Reply to this item at:
<http://savannah.gnu.org/bugs/?54567>
_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/
- [Octave-bug-tracker] [bug #54567] median(uint8) gives bad results, anonymous, 2018/08/24
- [Octave-bug-tracker] [bug #54567] median(uint8) gives bad results, Dan Sebald, 2018/08/24
- [Octave-bug-tracker] [bug #54567] median(uint8) gives bad results, Kai Torben Ohlhus, 2018/08/24
- [Octave-bug-tracker] [bug #54567] median(uint8) gives bad results, Kai Torben Ohlhus, 2018/08/24
- [Octave-bug-tracker] [bug #54567] median(uint8) gives bad results, Dan Sebald, 2018/08/24
- [Octave-bug-tracker] [bug #54567] median(uint8) gives bad results, Dan Sebald, 2018/08/24
- [Octave-bug-tracker] [bug #54567] median(uint8) gives bad results,
Kai Torben Ohlhus <=
- [Octave-bug-tracker] [bug #54567] median(uint8) gives bad results, Michael Godfrey, 2018/08/25
- [Octave-bug-tracker] [bug #54567] median(uint8) gives bad results, Dan Sebald, 2018/08/25
- [Octave-bug-tracker] [bug #54567] median(uint8) gives bad results, Rik, 2018/08/25
- [Octave-bug-tracker] [bug #54567] median(uint8) gives bad results, Dan Sebald, 2018/08/26
- [Octave-bug-tracker] [bug #54567] median(uint8) gives bad results, Dan Sebald, 2018/08/26
- [Octave-bug-tracker] [bug #54567] median(uint8) gives bad results, Dan Sebald, 2018/08/26
- [Octave-bug-tracker] [bug #54567] median(uint8) gives bad results, Dan Sebald, 2018/08/26
- [Octave-bug-tracker] [bug #54567] median(uint8) gives bad results, Michael Godfrey, 2018/08/26
- [Octave-bug-tracker] [bug #54567] median(uint8) gives bad results, Dan Sebald, 2018/08/26
- [Octave-bug-tracker] [bug #54567] median(uint8) gives bad results, Michael Godfrey, 2018/08/26