[Octave-bug-tracker] [bug #53456] output_precision: Cannot set precision higher than 16

[Oliver Heimlich]

Follow-up Comment #7, bug #53456 (project octave):

I agree that a limit on output_precision is of little use if it is used for
both single and double precision numbers: If it is forbidden (or discouraged)
to set precision higher than 16 for double, then 7 digits should be handled
equally for single precision. OTOH, if output_precision is used for built-in
floating-point formats only and output precision will be capped anyway
(starting with Octave 4.4), it is of no use to set a value higher than 16.

I assume that the only reason for capping the output precisions of both number
formats has been for Matlab compatibility of the “format long” command.

Thus, I'd like to suggest the following improvement (which would resolve the
regression): Internally, there should be two output precisions, one for single
and one for double. The output_precision function should have the following
syntaxes:

 · output_precision () or output_precision ("double") return double output
precision
 · output_precision ("single") return single output precision
 · output_precision (p, "single") set single output precision
 · output_precision (p, "double") set double output precision
 · output_precision (p) set both single and double output precision (like in
Octave 4.2 and earlier)

The number of digits displayed should never be capped (to 7 or 16 digits
respectively). Instead, for Matlab compatibility the “format long” command
should set each internal precision accordingly. “format short” (the
default format) should set both precisions to 5.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?53456>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/