|
| From: | Dmitri A. Sergatskov |
| Subject: | [Octave-bug-tracker] [bug #63848] Mean: some single class inputs can hit precision limit with sum |
| Date: | Wed, 1 Mar 2023 10:43:54 -0500 (EST) |
Follow-up Comment #9, bug #63848 (project octave):
It is a saturation at flintmax('single') = 16777216.
With double it is 9007199254740992
There are better summations algorithms.
See lots of discussions and proposals in
https://savannah.gnu.org/bugs/?61143
Dmitri.
--
_______________________________________________________
Reply to this item at:
<https://savannah.gnu.org/bugs/?63848>
_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/
| [Prev in Thread] | Current Thread | [Next in Thread] |