[Octave-bug-tracker] [bug #57926] Error dividing uint64 uint32 uint16 uint8 variables

[Hans]

URL:
  <https://savannah.gnu.org/bugs/?57926>

                 Summary: Error dividing uint64 uint32 uint16 uint8 variables
                 Project: GNU Octave
            Submitted by: alzi123
            Submitted on: Sat 29 Feb 2020 11:32:55 AM UTC
                Category: Octave Function
                Severity: 3 - Normal
                Priority: 5 - Normal
              Item Group: Incorrect Result
                  Status: None
             Assigned to: None
         Originator Name: alzi123
        Originator Email: 
             Open/Closed: Open
                 Release: 5.2.0
         Discussion Lock: Any
        Operating System: Microsoft Windows

    _______________________________________________________

Details:


I found an error in the division of the uintXX variable class.
A division by 2 does not correspond to a bit shift anymore due to some kind of
rounding error.

uint64(2^16-1)./uint64(2)  => 32768 and NOT 32767 due to rounding up instead
of rounding to the next lower number
uint32(2^16-1)./uint32(2)  => 32768 and NOT 32767 
uint16(2^16-1)./uint16(2)  => 32768 and NOT 32767 
double(2^16-1)./double(2) => 32767.5


A workaround would be the conversion to double before the devision: 

e.g.

a=uint64(2^16-1); 
b=uint64(2);
uint64(floor( double(a)./double(b) )) => 32767






    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?57926>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/