[Octave-bug-tracker] [bug #54572] int64 does not saturate correctly in negative direction

[Rik]

Follow-up Comment #18, bug #54572 (project octave):

I ran the code through the debugger and printed out the intermediate values.


(gdb) p x
$13 = -9223372036854775808
(gdb) p y
$14 = <optimized out>
(gdb) p ux
$15 = -2
(gdb) p uy
$16 = -9223372036854775808
(gdb) p (ux & uy)
$17 = -9223372036854775808
(gdb) p __signbit (~u)
$18 = 128


The code path that is taken ends up going through the if at the very bottom


    if ((ux & uy) < 0)
      {
        u = octave_int_base<T>::max_val () + __signbit (~u);
      }


I compiled again with -g -O1 and verified that the output is correct.  When I
run it under gdb the values are identical.  This points to the actual '+'
operator on the line above.


(gdb) p x
$1 = -9223372036854775808
(gdb) p y
$2 = <optimized out>
(gdb) p ux
$3 = -2
(gdb) p uy
$4 = -9223372036854775808
(gdb) n
136             u = octave_int_base<T>::max_val () + __signbit (~u);
(gdb) p ux & uy
$5 = -9223372036854775808
(gdb) p __signbit (~u)
$6 = 128





    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?54572>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/