[Octave-bug-tracker] [bug #36133] num2str displays more than 16 significant digits for large integer inputs

[Rik]

Follow-up Comment #8, bug #36133 (project octave):

Adding in jwe because the behavior is fascinating.  From Hartmut's tests in
comment #3, the Matlab sprintf function returns


sprintf ('%30.0f', 1e25)
ans =
    10000000000000001000000000


where the '1' in the sectond position is very real.

Octave's answer for the same code is


ans =     10000000000000000905969664


If I align the two


    10000000000000001000000000     # Matlab
    10000000000000000905969664     # Octave


it seems pretty clear what Matlab has done.  First, Octave uses the C library
routine for sprintf so we get whatever default behavior is implemented when
the precision exceeds that of the underlying number.  But Matlab has written a
shim layer for sprintf which calls the sprintf library, but then cleans up the
result.  However, they haven't been too clever about it.  IEEE-854 doubles
have a variable precision of 15-17 decimal digits.  The extra '1' occurs
because they are rounding the digit '9' which is the 18 significant figure and
not valid.

In any case, once Matlab has printed 17 digits, it pads the rest of the string
with zeros.  This is pretty easy to do in C++ with the substr() function or
operator[] access and a for loop.

So should Octave do something similar?  And if so, where should it be done? 
It could be done just in our sprintf routine, but then other actions like
printf would diverge in behavior.  That means it would probably be better to
do it in octave::stream.



    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?36133>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/