Corrupted display changing screen colour depth in qemu-system-ppc/MacOS

[Mark Cave-Ayland]

Hi all,

Howard pointed me off-list to a corrupted screen display issue experienced when 
changing the screen colour depth in MacOS under qemu-system-ppc. I was able to 
reproduce it here, and noticed from the output that the issue was likely due to the 
host display not updating its depth accordingly.

After it was confirmed to me that this was working in QEMU 6.2, I was able to 
eventually bisect the problem down to this commit:


cb8962c146b2633a4b04562281de9b2703bba849 is the first bad commit
commit cb8962c146b2633a4b04562281de9b2703bba849
Author: Marc-André Lureau <marcandre.lureau@redhat.com>
Date:   Tue Feb 15 00:13:37 2022 +0400

    ui: do not create a surface when resizing a GL scanout

    qemu_console_resize() will create a blank surface and replace the
    current scanout with it if called while the current scanout is
    GL (texture or dmabuf).

    This is not only very costly, but also can produce glitches on the
    display/listener side.

    Instead, compare the current console size with the fitting console
    functions, which also works when the scanout is GL.

    Note: there might be still an unnecessary surface creation on calling
    qemu_console_resize() when the size is actually changing, but display
    backends currently rely on DisplaySurface details during
    dpy_gfx_switch() to handle various resize aspects. We would need more
    refactoring to handle resize without DisplaySurface, this is left for a
    future improvement.

    Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
    Message-Id: <20220214201337.1814787-4-marcandre.lureau@redhat.com>
    Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>

 ui/console.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)


Some more background: the screen in qemu-system-ppc's MacOS is controlled via a 
custom driver written by Ben which uses the Bochs VBE registers to change the screen 
width/height/depth. The code used to do this can be found at 
https://gitlab.com/qemu-project/QemuMacDrivers/-/blob/master/QemuVGADriver/src/QemuVga.c#L354.

Looking at the changes in cb8962c146 my guess would be that either the updated check 
in qemu_console_resize() should also check to see if the surface depth is unchanged 
before exiting early, or that there is an extra update required in the VGA device 
when changing just the screen colour depth by itself.


ATB,

Mark.