[Octave-bug-tracker] [bug #54405] octave_idx_type index integer overflow math check doesn't work correctly

[Dan Sebald]

Follow-up Comment #3, bug #54405 (project octave):

Something else I just remembered: I think that both gcc and VCC have integer
math overflow trap mechanism (-trapv in gcc or something similar).

Also, GNU has some builtin routines that will check for overflow:

https://gcc.gnu.org/onlinedocs/gcc/Integer-Overflow-Builtins.html

These might be more efficient than the


    // Check for overflow.
    if (nr != 0 && abs (nc) > abs (std::numeric_limits<octave_idx_type>::max
() / nr))
      {
std::cerr << "ERROR OCCURED\n";
        error ("fread: dimension too large for Octave's index type");


approach because that division above could be wasteful for small matrix sizes.
 That is, C/C++ has no natural language elements for checking overflows, but
GNU might have optimized routines at the machine code level that can easily
check a status flag after the multiplication is done.

But Octave aims for all different platforms in a general sense, so using the
traps or the GNU 


bool __builtin_smul_overflow (int a, int b, int *res)


is sort of out.

Otherwise, is there a more efficient pre-multiply check?


log2(nc) > log2(std::numeric_limits<octave_idx_type>::max ()) - log2(nr)


perhaps?  The log2(MAX) part is just a constant.  log2 might have a really
efficient integer-based counterpart.

Or, one could do


if ((nr > 2^32 || nc > 2^32) && nr != 0 
    && nc > (std::numeric_limits<octave_idx_type>::max () / nr))
  {
    error ("fread: dimension too large");


Meaning that at least one of the nr and nc has to be greater than
2^(log2(MAX)/2) otherwise there will be no multiplication overflow.  That is a
very quick way of allowing a very large percentage of the most typical vector
sizes, i.e., fairly small.  Only when the dimension sizes start to get big
would a division be done to check for overflow.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?54405>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/