[Octave-bug-tracker] [bug #53166] format long displays 15 significant digits, rather than 15 digits after decimal point (Matlab Compatibility)

[Michael Leitner]

Follow-up Comment #3, bug #53166 (project octave):

That has to be either a Matlab bug or a Matlab documentation bug (I just
looked it up, the really talk about "after the decimal point"). Does that mean
that Matlab would print a 26-digit output for the following code, or do they
immediately switch to scientific notation for values larger or equal to 10?


format long
pi*10000000000


There is perhaps some very small Octave documentation inaccuracy in terms of
how "significant figures" are defined. Consider


1+2^-52
1-2^-52


Note that in the second case 15 zeros are printed, in contrast to only 14 in
the first case. This is because 1-2^-52 is actually 0.9999..., so the first
significant figure is the one after the decimal point, while it is before the
point for 1+2^-52. With rounding, both numbers look equal, so this distinction
is lost. But I see no reason to change the behaviour or to reword the
documentation to reflect that. 

But I think that Octave could do better (perhaps that is what Matlab actually
does without being clear about it), in the sense of "format long" implying to
output as much information as possible, that is as much figures as possible,
without ever faking information. eps(1)=2.2e-16, therefore in the range [1,2)
you can find for every fixed-point number with 15 digits after the decimal
point an IEEE 754-double that if rounded to those 15 digits gives exactly this
fixed-point number. In the range [2,4) your discretization is 4.4e-16, and for
[4,8) 8.8e-16. That is, if you print in this range 15 digits after the point,
you do not pretend any information that is not there (and this covers the case
of pi). Only for [8,10) you have a discretization of 1.8e-15, that is, here
there are some fixed-point numbers that you cannot represent as rounded
doubles, so you should print only 14 figures after the point (that is, the
present choice of 15 significant figures overall). 

By the way, during testing I found the following on my 4.0.3 Octave:


/tmp:162> a=(1-3*(2^-53))
a =  1.000000000000000
/tmp:163> a-.9
ans =  0.0999999999999996
/tmp:164> a-.99
ans =  0.00999999999999968


Now that is a bug in the second line due to incorrect rounding, or not?

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?53166>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/