[Octave-bug-tracker] [bug #53167] format long doesn't adjust precision to 7 digits when displaying single() values

[Rik]

Follow-up Comment #4, bug #53167 (project octave):

> Currently, a side effect of "format long" and "format short" is to set
output_precision. But that's not specific to single or double values. So we
have a choice or two to make.
> 
> Should we introduce a separate output_precision variable for single
precision values and have format long/short set both of those values?

That would be one way to handle it.

> 
> If not, then should we limit the display of values to the maximum number of
digits that double and single values can represent
(std::numeric_limits<T>::digits10 in C++)?

Is this really worth it?  The values aren't going to change given that we use
IEEE standards for single and double, so why not specify the value we want? 
I'm concerned that for double there are 15-17 significant figures.  Would
std::numeric_limits<T>::digits10 change based on the actual value?  That would
be weird.  We want a fixed value and it should probably be 16.  Given that we
know what we want it doesn't seem like we need to query std::numeric_limits. 

> 
> If we have a limit, should that limit apply to both single and double
precision values, and should it be imposed on the value of the
output_precision variable(s) or just as a limit inside the print functions?

???  I don't think I understand.  Doubles should be displayed with 16
significant figures and singles with 8 so there should be different limits.

> 
> Should we eliminate the output_precision and format_max_field_width
variables and just use the format function to set long/short/whatever format
styles?

This makes more sense since it is the value itself, in combination with the
format, which determines how it is displayed.  For example, floating point
numbers which are true integers ignore the output_precision argument.  It
seems reasonable that the underlying value, be it double or single, determines
how many digits are displayed rather than just a global variable.

> 
> None of these changes would affect *printf, so you could always do something
like "fprintf ('%.200f\n', val)" if you want to see the digits that the
floating point conversion routine generates when you ask for more precision
than is actually available.

There is a bug report somewhere about one of the conversion routines
generating extra nonsense digits after the first 16.  Apparently Matlab
doesn't do this.  For example,


sprintf ('%30.30d\n', 1e26)
ans =    100000000000000004764729344


This should just be a '1' with 26 '0's after it.


    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?53167>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/