[Octave-bug-tracker] [bug #31402] dec2base gives incorrect results for large integers.

[Gunnar Farnebäck]

URL:
  <http://savannah.gnu.org/bugs/?31402>

                 Summary: dec2base gives incorrect results for large
integers.
                 Project: GNU Octave
            Submitted by: gf
            Submitted on: Thu 21 Oct 2010 10:52:18 PM CEST
                Category: None
                Severity: 3 - Normal
                Priority: 5 - Normal
              Item Group: Incorrect Result
                  Status: None
             Assigned to: None
         Originator Name: Gunnar Farnebäck
        Originator Email: 
             Open/Closed: Open
         Discussion Lock: Any
                 Release: dev
        Operating System: GNU/Linux

    _______________________________________________________

Details:

Large integers which lose precision in a double may be converted incorrectly
by dec2base. E.g.

octave:72> dec2base(uint64(2)^63-1,16)
ans = 8GGFFFFFFFFFFFFF

The attached patch modifies the conversion to use a more robust algorithm,
giving the correct result

octave:73> dec2base(uint64(2)^63-1,16)
ans = 7FFFFFFFFFFFFFFF




    _______________________________________________________

File Attachments:


-------------------------------------------------------
Date: Thu 21 Oct 2010 10:52:18 PM CEST  Name: large_integers_in_dec2base.diff
 Size: 1kB   By: gf

<http://savannah.gnu.org/bugs/download.php?file_id=21752>

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?31402>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/