[Octave-bug-tracker] [bug #56936] format short displays 6, rather than 5, significant digits as documented

[Robert Jenssen]

Follow-up Comment #5, bug #56936 (project octave):

I know it is pointless saying that MATLAB is wrong but in this case MATLAB is
wrong! Left-hand zeros are not significant. The correct result to 5
significant figures is "0.33333". For the default format short,
output_precision(5), the number of significant digits shown varies between 4
and 7:

octave:1> version
ans = 7.0.0
octave:2> format short
octave:3> output_precision
ans = 5
octave:4> 0.00123456
ans = 1.2346e-03
octave:5> 0.0123456
ans = 0.012346
octave:6> 0.123456
ans = 0.1235
octave:7> 1.23456
ans = 1.2346
octave:8> 12.3456
ans = 12.346
octave:9> 123.456
ans = 123.46
octave:10> 1234.56
ans = 1234.6
octave:11> 12345.6
ans = 1.2346e+04
octave:12> 123456
ans = 123456
octave:13> 1234567
ans = 1234567
octave:14> 12345678
ans = 1.2346e+07


    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?56936>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/