[Octave-bug-tracker] [bug #63940] `format native-bit` gives wrong results

[Arun Giridhar]

Update of bug #63940 (project octave):

                  Status:         Patch Submitted => In Progress            

    _______________________________________________________

Follow-up Comment #3:

@nrjank: Thanks for checking. Could you also check Matlab for what it does for
these?

>> format hex; uint16 (45678)
>> format hex;  int16 (32001)

Octave gives b26e and 7d01 respectively. The equivalent for `bit` matches that
of `hex` but `native-bit` does not match `native-hex` for integer inputs and
learning Matlab's output will give a target.

I expanded the existing diff into a full patch with more documentation. Since
this is addressing an internal inconsistency in Octave, I pushed it to stable
here: https://hg.savannah.gnu.org/hgweb/octave/rev/d4d3098a15dc. It's correct
for floating point. Based on Matlab's integer behavior in the tests above, it
may have to be tuned a bit.

This is correct behavior from unpatched Octave (it matches the documentation
for hex, native-hex, and bit):

octave:2> format hex; pi
ans = 400921fb54442d18

octave:3> format native-hex; pi
ans = 182d4454fb210940

octave:4> format bit; pi
ans = 0100000000001001001000011111101101010100010001000010110100011000


This last one is incorrect without the patch (it is inconsistent with Octave's
own native-hex):

octave:5> format native-bit; pi
ans = 0001100010110100001000100010101011011111100001001001000000000010


To be consistent with 182d4454fb210940, it should give 0001 1000 0010 1101
0100 0100 0101 0100 1111 1011 0010 0001 0000 1001 0100 0000, but instead it
gives 0001 1000 1011 0100 0010 0010 0010 1010 1101 1111 1000 0100 1001 0000
0000 0010 which is equivalent to 18b4222adf849002 in hex, not matching
anything.

With the patch it gives the correct behavior:

octave:1> format hex; pi
ans = 400921fb54442d18

octave:2> format native-hex; pi
ans = 182d4454fb210940

octave:3> format bit; pi
ans = 0100000000001001001000011111101101010100010001000010110100011000

octave:4> format native-bit; pi
ans = 0001100000101101010001000101010011111011001000010000100101000000




    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?63940>

_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/