[Octave-bug-tracker] [bug #53683] numerical output format for float values with magnitude greater than precision is wrong

[Dan Sebald]

URL:
  <http://savannah.gnu.org/bugs/?53683>

                 Summary: numerical output format for float values with
magnitude greater than precision is wrong
                 Project: GNU Octave
            Submitted by: sebald
            Submitted on: Wed 18 Apr 2018 12:46:11 AM UTC
                Category: Interpreter
                Severity: 3 - Normal
                Priority: 5 - Normal
              Item Group: Incorrect Result
                  Status: None
             Assigned to: None
         Originator Name: 
        Originator Email: 
             Open/Closed: Open
         Discussion Lock: Any
                 Release: dev
        Operating System: Any

    _______________________________________________________

Details:

It caught my attention that the output format is displaying too much precision
for numbers that have magnitude greater than the number of digits of
precision, stated imprecisely.  That is, notice the progression:


octave:1> for i=-2:8
>   10^i / 3
> endfor
ans =  0.0033333
ans =  0.033333
ans =  0.33333
ans =  3.3333
ans =  33.333
ans =  333.33
ans =  3333.3
ans =  33333.33333
ans =  333333.33333
ans =  3333333.33333
ans =  33333333.33333


that once the left digits hits 5 the right digits jumps back up to five.

Here are some related bug reports, but they aren't quite the same issue:

https://savannah.gnu.org/bugs/?func=detailitem&item_id=53456
https://savannah.gnu.org/bugs/?func=detailitem&item_id=41359




    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?53683>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/