[Octave-bug-tracker] [bug #41359] Output formatting inconsistency for integers > 2^52

[Mike Miller]

Update of bug #41359 (project octave):

                  Status:                    None => Confirmed              
                 Release:                   3.6.4 => dev                    

    _______________________________________________________

Follow-up Comment #1:

Yes, you are right that the values should be representable in 16 decimal
digits. The basis of the decision whether a number can be displayed as an
integer or not is failing for these cases. The output formatter then believes
that the value is not strictly an integer and displays what you are seeing.

Here are essentially the computations that the formatter is doing to determine
whether the value is an integer or not:


> a = 2^52;
> a == floor (a + .5)
ans =  1
> a-1 == floor (a-1 + .5)
ans =  1
> a+1 == floor (a+1 + .5)
ans = 0
> a+2 == floor (a+2 + .5)
ans =  1


Since we've hit the limit of the significand, adding 0.5 rolls over into the
next bit position, floor gives the wrong result, and it appears that a+1 is
not an integer according to this formula.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?41359>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/