[Octave-bug-tracker] [bug #63230] Docking Command Window to the right side crashes Octave GUI and resets the GUI

[Markus Mützel]

Follow-up Comment #6, bug #63230 (project octave):

Thanks. Looks like gathering the backtraces worked.

I think the most important part might be this:

Thread 1 (Thread 30648.0x4f20):
#0  0x00007ff9dff6f633 in ntdll!RtlIsZeroMemory () from
C:\WINDOWS\SYSTEM32\ntdll.dll
#1  0x00007ff9dff783f2 in ntdll!RtlpNtSetValueKey () from
C:\WINDOWS\SYSTEM32\ntdll.dll
#2  0x00007ff9dff786da in ntdll!RtlpNtSetValueKey () from
C:\WINDOWS\SYSTEM32\ntdll.dll
#3  0x00007ff9dff7e361 in ntdll!RtlpNtSetValueKey () from
C:\WINDOWS\SYSTEM32\ntdll.dll
#4  0x00007ff9dfe960e5 in ntdll!RtlGetCurrentServiceSessionId () from
C:\WINDOWS\SYSTEM32\ntdll.dll
#5  0x00007ff9dfe95b74 in ntdll!RtlGetCurrentServiceSessionId () from
C:\WINDOWS\SYSTEM32\ntdll.dll
#6  0x00007ff9dfe947b1 in ntdll!RtlFreeHeap () from
C:\WINDOWS\SYSTEM32\ntdll.dll
#7  0x00007ff9def49c9c in msvcrt!free () from C:\WINDOWS\System32\msvcrt.dll
#8  0x0000000000862f6b in
liboctgui-8!_ZN15QConsolePrivate21syncConsoleParametersEv () from
D:\Software\Octave-7.2.0\mingw64\bin\liboctgui-8.dll
#9  0x00000000008634ae in
liboctgui-8!_ZN15QConsolePrivate17updateConsoleSizeEbb () from
D:\Software\Octave-7.2.0\mingw64\bin\liboctgui-8.dll
#10 0x00000000009dbe2d in
liboctgui-8!_ZN12QConsoleView11resizeEventEP12QResizeEvent () from
D:\Software\Octave-7.2.0\mingw64\bin\liboctgui-8.dll


IIUC, that might mean that we are trying to free memory twice.
The last symbol in an Octave library demangles to
`QConsolePrivate::syncConsoleParameters()`. Its implementation is here:
https://hg.savannah.gnu.org/hgweb/octave/file/42990806eb9d/libgui/qterminal/libqterminal/win32/QWinTerminalImpl.cpp#l1052

The part where memory is freed in that function is:

  if (m_buffer)
    delete [] m_buffer;
  if (m_tmpBuffer)
    delete [] m_tmpBuffer;


I can't see where those buffers are freed anywhere else (apart from the
destructor of QConsolePrivate).

A few lines above, it looks like the console buffer size is adjusted
separately for changes in height or in width. Maybe that is assuming that it
is only changing one or the other in one step. But when redocking, both
directions change at the same time. Does that make sense? Could that lead to a
crash?

Couldn't we do that in one go? E.g., like so:

diff -r 1c4e017664fd libgui/qterminal/libqterminal/win32/QWinTerminalImpl.cpp
--- a/libgui/qterminal/libqterminal/win32/QWinTerminalImpl.cpp  Sat Oct 22
13:33:54 2022 +0200
+++ b/libgui/qterminal/libqterminal/win32/QWinTerminalImpl.cpp  Sat Oct 22
14:14:43 2022 +0200
@@ -371,7 +371,8 @@
   m_font.setPointSize (9);
   m_font.setStyleHint (QFont::TypeWriter);
 
-  m_buffer = m_tmpBuffer = 0;
+  m_buffer = nullptr;
+  m_tmpBuffer = nullptr;
 
   m_consoleView = new QConsoleView (parent);
   m_horizontalScrollBar = new QScrollBar (Qt::Horizontal, parent);
@@ -1066,22 +1067,11 @@
   sr.Top    = m_consoleRect.top ();
   sr.Bottom = m_consoleRect.bottom ();
 
-  if (bs.Y > sbi.dwSize.Y)
-    {
-      SetConsoleScreenBufferSize (hStdOut, bs);
-      SetConsoleWindowInfo (hStdOut, TRUE, &sr);
-    }
-  else
-    {
-      SetConsoleWindowInfo (hStdOut, TRUE, &sr);
-      SetConsoleScreenBufferSize (hStdOut, bs);
-    }
-
   bs.X = m_bufferSize.width ();
   sr.Left  = m_consoleRect.left ();
   sr.Right = m_consoleRect.right ();
 
-  if (bs.X > sbi.dwSize.X)
+  if (bs.Y * bs.X > sbi.dwSize.Y * sbi.dwSize.X)
     {
       SetConsoleScreenBufferSize (hStdOut, bs);
       SetConsoleWindowInfo (hStdOut, TRUE, &sr);


Anyone having any idea what might be happening here?



    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?63230>

_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/