[Octave-bug-tracker] [bug #53166] format long displays 15 significant digits, rather than 15 digits after decimal point (Matlab Compatibility)

[John W. Eaton]

Follow-up Comment #8, bug #53166 (project octave):

Ignoring for a moment whether 15 or 16 digits is correct, here is what Octave
will try to do now, using double precision values for example.  With format
short (the default), Octave will try to display at least 5 significant
figures, with 5 digits after the decimal and will allow the decimal position
to float.  The maximum number of significant digits displayed is currently 16
(maybe it should be 15, but that's a separate issue).  If that point is
reached, the display switches to E format.  There is also a maximum field
width.  If that is exceeded (say, by attempting to display a small fraction as
0.0000...NNNN) then the display switches to E format.  For arrays, you can get
results like this:


octave> n = 5; (logspace (-n, n, 2*n+1) * 1.2345678901234567890)'
ans =

        0.000012346
        0.000123457
        0.001234568
        0.012345679
        0.123456789
        1.234567890
       12.345678901
      123.456789012
     1234.567890123
    12345.678901235
   123456.789012346


Changing N to 6 in the expression above will give you short E format:


octave> n = 6; (logspace (-n, n, 2*n+1) * 1.2345678901234567890)'
ans =

   1.2346e-06
   1.2346e-05
   1.2346e-04
   1.2346e-03
   1.2346e-02
   1.2346e-01
   1.2346e+00
   1.2346e+01
   1.2346e+02
   1.2346e+03
   1.2346e+04
   1.2346e+05
   1.2346e+06


This seems quite reasonable to me.

If you prefer fixed format more like what Matlab provides, then you can use
"fixed_point_format (true)" and Octave will show you something like this:


octave> fixed_point_format (true); n = 5; (logspace (-n, n, 2*n+1) *
1.2345678901234567890)'
ans =

  1.0e+05 *

   0.00000
   0.00000
   0.00000
   0.00000
   0.00000
   0.00001
   0.00012
   0.00123
   0.01235
   0.12346
   1.23457


I believe this is (approximately) what Matlab does by default.  I find it less
useful in most cases than allowing the decimal position to float.

For scalar values, the rules are the same, but not as evident.  Instead of
what Matlab is doing, Octave will show you this:


octave> format long, pi, 10*pi, 100*pi
ans =  3.141592653589793
ans =  31.41592653589793
ans =  314.1592653589793


With fixed format, the results are similar.

With scalar values, I just noticed a small problem when writing this comment. 
Octave doesn't properly switch to E format when the number of digits to be
displayed exceeds the number of digits of precision available for the given
data type.  So it will do things like this:


format long, pi, 1e19*pi, 1e20*pi
ans =  3.141592653589793
ans =  31415926535897931776
ans =    3.141592653589793e+20


I'll take a look at fixing that now.


    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?53166>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/