|
From: | Hans |
Subject: | [Octave-bug-tracker] [bug #57926] Error dividing uint64 uint32 uint16 uint8 variables |
Date: | Sat, 29 Feb 2020 06:32:57 -0500 (EST) |
User-agent: | Mozilla/5.0 (Windows NT 6.1; WOW64; rv:56.0) Gecko/20100101 Firefox/56.0 |
URL: <https://savannah.gnu.org/bugs/?57926> Summary: Error dividing uint64 uint32 uint16 uint8 variables Project: GNU Octave Submitted by: alzi123 Submitted on: Sat 29 Feb 2020 11:32:55 AM UTC Category: Octave Function Severity: 3 - Normal Priority: 5 - Normal Item Group: Incorrect Result Status: None Assigned to: None Originator Name: alzi123 Originator Email: Open/Closed: Open Release: 5.2.0 Discussion Lock: Any Operating System: Microsoft Windows _______________________________________________________ Details: I found an error in the division of the uintXX variable class. A division by 2 does not correspond to a bit shift anymore due to some kind of rounding error. uint64(2^16-1)./uint64(2) => 32768 and NOT 32767 due to rounding up instead of rounding to the next lower number uint32(2^16-1)./uint32(2) => 32768 and NOT 32767 uint16(2^16-1)./uint16(2) => 32768 and NOT 32767 double(2^16-1)./double(2) => 32767.5 A workaround would be the conversion to double before the devision: e.g. a=uint64(2^16-1); b=uint64(2); uint64(floor( double(a)./double(b) )) => 32767 _______________________________________________________ Reply to this item at: <https://savannah.gnu.org/bugs/?57926> _______________________________________________ Message sent via Savannah https://savannah.gnu.org/
[Prev in Thread] | Current Thread | [Next in Thread] |