[Octave-bug-tracker] [bug #54572] int64 does not saturate correctly in negative direction

[Dan Sebald]

Follow-up Comment #32, bug #54572 (project octave):

@JWE: You modified both the add() and sub() routines to use sign comparison
rather than adding a value.  It looks to me that the purpose of that sign test
code is exactly the same in both cases.  So rather than this (the
__signbit(~u)):


  // This is very similar to addition.
  static T
  sub (T x, T y)
  {
#if defined (OCTAVE_HAVE_FAST_INT_OPS)
    // The typecasts do nothing, but they are here to prevent an optimizing
    // compiler from interfering.  Also, the signed operations on small types
    // actually return int.
    T u = static_cast<UT> (x) - static_cast<UT> (y);
    T ux = u ^ x;
    T uy = u ^ ~y;
    if ((ux & uy) < 0)
      {
        u = (__signbit (~u)
             ? octave_int_base<T>::min_val ()
             : octave_int_base<T>::max_val ());
      }


which uses the 1's complement, how about making them consistent?  I.e.,
__signbit(~u) = !__signbit(u) = !(u < 0), so switch around the ternary
operator arguments and get


        u = (u < 0
             ? octave_int_base<T>::max_val ()
             : octave_int_base<T>::min_val ());


the same as for add().  In some sense, that last portion of the routine is
independent of the modulo arithmetic that took place before it.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?54572>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/