[Octave-bug-tracker] [bug #54572] int64 does not saturate correctly in negative direction

[John W. Eaton]

Follow-up Comment #20, bug #54572 (project octave):

Ah, instead of


      u = octave_int_base<T>::max_val () + __signbit (~u);


I think we can write:


      u = (__signbit (~u)
           ? std::numeric_limits<T>::min ()
           : std::numeric_limits<T>::max ());


because it seems as though the intent was to either set the result to the max
value for the type (signbit is 0) or, if signbit is 1, add 1 to the max value
and wrap around to the min value.  Why not set the min and max directly,
simply by checking the sign?

Also, I think octave_int_base<T>::max_val () was the wrong thing here anyway,
because if I'm reading the code correctly, that returns an octave_int<T> type,
so wouldn't the operator + applied to that value send us back through all the
octave_int code for adding the signbit (0 or 1) value??

The original code seems like it tried to be a bit too tricky for simply
setting the max or min value if an overflow condition is detected.

I'm hereby coining a new term that may be used to describe this type of
problem in Octave:  Jaroslavimization.  :-)

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?54572>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/