[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Octave-bug-tracker] [bug #54405] octave_idx_type index integer overflow
From: |
Rik |
Subject: |
[Octave-bug-tracker] [bug #54405] octave_idx_type index integer overflow math check doesn't work correctly |
Date: |
Wed, 1 Aug 2018 00:05:41 -0400 (EDT) |
User-agent: |
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:61.0) Gecko/20100101 Firefox/61.0 |
Follow-up Comment #29, bug #54405 (project octave):
It turns out that we are catching some version of abs() which doesn't seem to
work very well. If I specify std::abs explicitly things work. See the
results of some printf debugging.
max: 9223372036854775807
nr: -1
nc: 1
max / nr: -9223372036854775807
abs (max / nr): 1
std::abs (max / nr): 9223372036854775807
But now that I know what is going on I can get around it by avoiding abs()
entirely. I re-coded to
// Check for overflow.
if (nr > 0 && nc > 0
&& nc > std::numeric_limits<octave_idx_type>::max () / nr)
::error ("%s: size too large for Octave's index type", who.c_str ());
I checked in this amendment here
(https://hg.savannah.gnu.org/hgweb/octave/rev/02fbbde15308).
_______________________________________________________
Reply to this item at:
<http://savannah.gnu.org/bugs/?54405>
_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/
- [Octave-bug-tracker] [bug #54405] octave_idx_type index integer overflow math check doesn't work correctly,
Rik <=