[Octave-bug-tracker] [bug #54405] octave_idx_type index integer overflow math check doesn't work correctly

[Rik]

Follow-up Comment #29, bug #54405 (project octave):

It turns out that we are catching some version of abs() which doesn't seem to
work very well.  If I specify std::abs explicitly things work.  See the
results of some printf debugging.


max: 9223372036854775807
nr: -1
nc: 1
max / nr: -9223372036854775807
abs (max / nr): 1
std::abs (max / nr): 9223372036854775807


But now that I know what is going on I can get around it by avoiding abs()
entirely.  I re-coded to


// Check for overflow.
if (nr > 0 && nc > 0
    && nc > std::numeric_limits<octave_idx_type>::max () / nr)
  ::error ("%s: size too large for Octave's index type", who.c_str ());


I checked in this amendment here
(https://hg.savannah.gnu.org/hgweb/octave/rev/02fbbde15308).


    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?54405>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/