[Octave-bug-tracker] [bug #36133] num2str displays more than 16 significant digits for large integer inputs

[Rik]

Follow-up Comment #10, bug #36133 (project octave):

Results from Matlab will be interesting, but I doubt that they test that each
number to be printed is an exact power of two.  There are 1024 of those
numbers compared to 2^53 (9e15) numbers representable in the mantissa.  It
really, really fails the 80/20 rule to worry about them.

According to Matlab's documentation for both sprintf and num2str, they are
aware of the issue


Note

If you specify a precision operator for floating-point values that exceeds the
precision of the input numeric data type, the results might not match the
input values to the precision you specified. The result depends on your
computer hardware and operating system.


Also, according to the documentation for num2str


s = num2str(A) converts a numeric array into a character array that represents
the numbers. The output format depends on the magnitudes of the original
values


That's fine, but my guess is that Matlab just uses a form of the %g format for
both floating point and integer numbers.  Octave, however, tries to switch
format based on whether the input is an integer.  The code is


if (any (x(valid) != fix (x(valid))))
  ## Floating point input
  ndgt = max (ndgt + 5, 5);   # Keep at least 5 significant digits
  ndgt = min (ndgt, 16);      # Cap significant digits at 16
  fmt = sprintf ("%%%d.%dg", ndgt+7, ndgt);
else
  ## Integer input
  ndgt += 3;
  if (any (! valid))
    ndgt = max (ndgt, 5);     # Allow space for Inf/NaN
  endif
  ## FIXME: Integers must be masked to show only 16 significant digits
  ##        See test case for bug #36133 below
  fmt = sprintf ("%%%d.0f", ndgt);
endif


I bet changing the integer case would fix some of this. 


    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?36133>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/