[Octave-bug-tracker] [bug #47914] segfault with OpenGL patches and address sanitizer

[Rik]

Follow-up Comment #19, bug #47914 (project octave):

I'm pretty certain now this is a bug in the SW renderer.  I ran the
development version recently and the stack trace showed that the problem was
with disabling clip planes.


  opengl_renderer::set_clipping (bool enable)
  {
#if defined (HAVE_OPENGL)

    bool has_clipping = (glIsEnabled (GL_CLIP_PLANE0) == GL_TRUE);

    if (enable != has_clipping)
      {
        if (enable)
          for (int i = 0; i < 6; i++)
            glEnable (GL_CLIP_PLANE0+i);
        else
          for (int i = 0; i < 6; i++)
            glDisable (GL_CLIP_PLANE0+i);
      }


I used printf debug statements to find that it was the very first call to


  glDisable (GL_CLIP_PLANE0+i);  // i = 0


which was causing the heap-buffer-overflow.

I get this only when I have set LIBGL_ALWAYS_SOFTWARE to 1.  If I unset this
environment variable then the ASAN error goes away.




    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?47914>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/