[Octave-bug-tracker] [bug #58147] Matlab 2020 compatibility: dec2bin, dec2hex shouldn't error on negative inputs

[Rik]

Follow-up Comment #12, bug #58147 (project octave):

The current Octave implementation has a problem with numbers greather than
flintmax().  We should either restrict the range of inputs and issue an error
if the input is outside that range, or modify the algorithm.

A test case is


dec2bin (-2^31 -1)


which should return 


"1111111111111111111111111111111101111111111111111111111111111111"


, but instead returns


"1111111111111111111111111111111110000000000000000000000000000000"


The trouble is this line, I think


d(lt_zero_idx) += double (intmax ("uint64")) + 1;


intmax() for "uint64" is larger than flintmax() (2^53) for double values and
so precision is lost.



    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?58147>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/