[Octave-bug-tracker] [bug #65495] nchoosek error "gcd: all values must be integers"

[anonymous]

Follow-up Comment #14, bug #65495 (group octave):


> b)
> The other issue is that the gcd function does NOT always provide a correct
result for floating point numbers beyond flintmax.
> Example:
> gcd (uint64(879236959347161415), 9)
> -> 9
> gcd (double(879236959347161415), 9)
> -> 3

`double(879236959347161415)` is the same as `879236959347161415` which has an
actual value of 879236959347161472.  The GCD of this and 9 is 3.

Side note:  I thought numeric literals were always doubles, which would make
`uint64(879236959347161415)` have an actual value of 879236959347161472, but I
just checked in Octave 8.4.0 and it seems to have an actual value of
879236959347161415.  I don't recall seeing mention of this in the
documentation.



    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?65495>

_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/