[Octave-bug-tracker] [bug #53644] Possible Octave or Nouveau image plot bug (not sure which) discovered via ASAN

[Dan Sebald]

Follow-up Comment #6, bug #53644 (project octave):

Found it.  Whew, that one was a "doozy" (using mid-20th century idiom).  It's
a bug in plain view, but until one realizes how it comes about, it's
baffling.

So, the main clue is that the opengl_renderer::draw_axes routine fails in the
same code line every time, but it isn't until the second invocation of
"image()" that the ASAN abort occurs.  Look at the following code and notice
there are two instances of draw_axes_boxes()


  void
  opengl_renderer::draw_axes (const axes::properties& props)
  {
[snip]
    setup_opengl_transformation (props);

    // For 2D axes with only 2D primitives, draw from back to front without
    // depth sorting
    bool is2D = props.get_is2D (true);
    if (is2D)
      glDisable (GL_DEPTH_TEST);
    else
      glEnable (GL_DEPTH_TEST);

    draw_axes_planes (props);

    if (! is2D || props.layer_is ("bottom"))
      {
        draw_axes_grids (props);
        if (props.get_tag () != "legend" || props.get_box () != "off")
          draw_axes_boxes (props);
      }

[this line disables the glLineStipple activated by draw_axes_boxes(), but...]
    set_linestyle ("-");  // Disable LineStipple

    set_clipbox (x_min, x_max, y_min, y_max, z_min, z_max);

    draw_axes_children (props);

    if (is2D && props.layer_is ("top"))
      {
        draw_axes_grids (props);
        if (props.get_tag () != "legend" || props.get_box () != "off")
          draw_axes_boxes (props);
      }

[...but there is nothing here deactivating the glLineStipple of the second
draw_axes_boxes()]


Basically, there was nothing disabling any possibly-set glLineStipple in the
draw_axes_boxes() routine such that on the second invocation of image() the
stipple was set coming into this routine.  The draw_axes_planes() routine uses
GL_QUADS, but filled quads have a line option.  I suppose having stipple on
with those quads causes Nouveau to look for stipple information just outside
the valid memory region for the stored quad.  That is, in this ASAN output:


Shadow bytes around the buggy address:
  0x0c22801a0080: fd fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c22801a0090: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c22801a00a0: fd fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa
  0x0c22801a00b0: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x0c22801a00c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c22801a00d0: 00 00 00 00 00[fa]fa fa fa fa fa fa fa fa fa fa
  0x0c22801a00e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c22801a00f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c22801a0100: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c22801a0110: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c22801a0120: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa


the [fa] is the out-of-range address access (00 means addressable) just after
what is probably the quad stored in memory.  One could say this may be a
Nouveau issue because it should be able to address the situation when stipple
is on, but as I see it we really aren't using such a scenario so I think we
can avoid it.  Plus, it makes good sense to have every instance of setting
glLineStipple balanced with turning it off after line-drawing is done.

Attached is a patch that makes the ASAN dump go away, and it also makes the
documentation figure failures go away on KDE/Plasma where I'm using Nouveau
driver and not Nvidia driver.


(file #43931)
    _______________________________________________________

Additional Item Attachment:

File name:
octave-opengl_qt_toolkit_line_stipple_nouveau_driver-djs2018apr15.patch Size:1
KB


    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?53644>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/