[Octave-bug-tracker] [bug #54572] int64 does not saturate correctly in negative direction

[Dan Sebald]

Follow-up Comment #28, bug #54572 (project octave):

Looks good Rik.  There may be a few other corner cases to add, however, such
as in this case:


octave:4> intmin ('int64') + intmin ('int64')
ans = -9223372036854775808


The reason is that in all the fast algorithms (be it the __builtin_X group or
the whiz-bang version), the direction of wrap has to be deduced from the
operation result.  For example, in the patch I had something like:


    if (__builtin_saddll_overflow (x, y, &u))
      {
        if (u < 0)
          u = octave_int_base<T>::max_val ();


being careful with the "< 0" comparison, not "<= 0".  Think about the integer
represenation


[-N ... -1][0 ... N-1]


If x = -N and y = -N, the addition (or subtraction for that matter) operation
results in u=0.

In the case of unsigned, I suppose it's more straightforward.  Since there are
no negative numbers, I suppose addition can only overflow at INT_MAX and
subtraction can only overflow at INT_MIN.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?54572>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/