[no subject]

[Phil Dennis-Jordan]

Date: Mon, 15 Jul 2024 21:07:12 +0200
Subject: [PATCH 00/26] hw/display/apple-gfx: New macOS PV Graphics device
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

This sequence of patches integrates the paravirtualised graphics device
implemented by macOS's ParavirtualizedGraphics.Framework into Qemu.
Combined with the guest drivers which ship with macOS versions 11 and up,
this allows the guest OS to use the host's GPU for hardware accelerated
3D graphics, GPGPU compute (both using the 'Metal' graphics API), and
window compositing.

Some background:
----------------

The device exposed by the ParavirtualizedGraphics.Framework's (henceforth
PVG) public API consists of a PCI device with a single memory-mapped BAR;
the VMM is expected to pass reads and writes through to the framework, and
to forward interrupts emenating from it to the guest VM.

The bulk of data exchange between host and guest occurs via shared memory,
however. For this purpose, PVG makes callbacks to VMM code for allocating,
mapping, unmapping, and deallocating "task" memory ranges. Each task
represents a contiguous host virtual address range, and PVG expects the
VMM to map specific guest system memory ranges to these host addresses via
subsequent map callbacks. Multiple tasks can exist at a time, each with
many mappings.

Data is exchanged via an undocumented, Apple-proprietary protocol. The
PVG API only acts as a facilitator for establishing the communication
mechanism. This is perhaps not ideal, and among other things means it
only works on macOS hosts, but it's the only serious option we've got for
good performance and quality graphics with macOS guests at this time.

The first iterations of this PVG integration into Qemu were developed
by Alexander Graf as part of his "vmapple" machine patch series for
supporting aarch64 macOS guests, and posted to qemu-devel in June and
August 2023:

https://lore.kernel.org/all/20230830161425.91946-1-graf@amazon.com/T/

This integration mimics the "vmapple"/"apple-gfx" variant of the PVG device
used by Apple's own VMM, Virtualization.framework. This variant does not use
PCI but acts as a direct MMIO system device; there are two MMIO ranges, one
behaving identically to the PCI BAR, while the other's functionality is
exposed by private APIs in the PVG framework. It is only available on aarch64
macOS hosts.

I had prior to this simultaneously and independently developed my own PVG
integration for Qemu using the public PCI device APIs, with x86-64 and
corresponding macOS guests and hosts as the target. After some months of
use in production, I was slowly reviewing the code and readying it for
upstreaming around the time Alexander posted his vmapple patches.

I ended up reviewing the vmapple PVG code in detail; I identified a number
of issues with it (mainly thanks to my prior trial-and-error working with
the framework) but overall I thought it a better basis for refinement
than my own version:

 - It implemented the vmapple variant of the device. I thought it better to
   port the part I understood well (PCI variant) to this than trying to port
   the part I didn't understand well (MMIO vmapple variant) to my own code.
 - The code was already tidier than my own.

It also became clear in out-of-band communication that Alexander would
probably not end up having the time to see the patch through to inclusion,
and was happy for me to start making changes and to integrate my PCI code.

It's taken a while, but I'm happy with the result and think it will be a
welcome addition for anyone running macOS VMs.

What doesn't work:
------------------

 * State (de-)serialisation and thus migration. There is no fundamental
   technical obstacle to this. PVG supports saving and loading device state.
   I have simply not had the resources to implement (and crucially, test it)
   it yet.
 * Setting the list of display modes via a property is currently only
   implemented on the PCI version, which is the only one readily testable
   without the out-of-tree vmapple patches. (See review notes for patch 24)
 * End-to-end GPU-only rendering. After the host GPU has rendered the guest's
   screen, the framebuffer is copied into a system memory buffer (surface).
   When using the Qemu Cocoa UI, this buffer is drawn by the CPU into a GPU
   texture used for hardware window compositing. It would be vastly more
   efficient to retain the Metal texture and pass it directly through to the
   Cocoa window. We currently have no mechanism for doing so; it would need
   to be similar to the end-to-end OpenGL rendering support, with the added
   complication that Metal textures are Objective-C types and would need to
   traverse the plain C code of the Qemu display subsystem.
 * Dirty region detection. Similarly, the whole framebuffer is marked modified
   even if there has only been a small change. This hurts network data volume
   when using VNC.
 * Multi-head support. PVG allows "connecting" more than one virtual display.
   This integration currently always uses exactly 1 display.
 * The vmapple/aarch64 variant of the device is only testable with Alexander's
   vmapple machine type patch set. I've been maintaining this out-of-tree and
   have made a few improvements, but it doesn't yet run smoothly. (Graphics
   work fine with this code, issues are with other devices.) I can push my
   current draft to a git forge if anyone wants to test with them. I'm
   definitely hoping to eventually resolve the remaining problems and submit
   a revised version of that patch set as well.


I think we can live without these for the moment, and I'd prefer to work on
them only if and when the baseline functionality has been merged.

Patch review notes:
-------------------

I have aimed to submit the patches roughly in order of descending importance
and increasing debatability. From patch 18 it becomes usable and useful in
practice without further modification. Patches 19-23 fix issues that only occur
in certain host configurations or that can otherwise be worked around. Patch
24 is more of an RFC; patch 26 is needed if recently submitted Cocoa UI patches
end up being merged.

Brief meta-discussion of specific patches and groups of patches:

 01:    I have retained Alexander Graf's original patch intact as far as
        possible as patch 01. My only modifications are those required for
        fixing rebase conflicts.
 02:    Resolves build errors caused by upstream API changes since original
        patch submission.
 03:    This moves the device code to hw/display from its original hw/vmapple.
        With the PCI variant added later, it doesn't make much sense to put
        this inside a machine type directory.
 04-13: These patches address issues identified during code review in the
        original e-mail threads as well as my own review.
 14-15: These patches split the monolithic source file into a common core and
        vmapple/mmio specific parts.
        NOTE: checkpatch.pl complains about #ifdef __OBJC__ in the header file.
        This is needed for allowing the file to be #included from pure C.
        Ensuring that isn't currently critical, but I expect it to be useful
        in future.
 16-17: Preparation for x86-64. The x86-64 variant of PVG seems to behave
        slightly differently than the aarch64 version in terms of threading
        and control flow edge cases. We need to do some things asynchronously
        to avoid deadlock and performance problems.
 18:    The PCI variant. This builds on the split from 14/15 and the async
        changes to create the PVG PCI device which works with x86-64 macOS
        guests.
 19-23: Fixes for various additional problems and limitations. Without these
        the PVG device will only work with the Cocoa UI, on Macs which
        have an integrated (shared memory) GPU only, and mouse cursors will
        be slightly off, for example.
 24:    QOM property for specifying the display mode list (resolutions) the
        device will report to the guest. I checked other display devices and
        found none supported this, though I personally find it very useful.
        I'm wondering whether this should be a more generic feature optionally
        usable by any display device in Qemu?
 25:    Adding myself as maintainer for the PVG code, and reviewer for HVF.
 26:    Required if/when Akihiko Odaki's pending patch for removing
        dpy_cursor_define_supported is merged.

I don't know if it's useful/wise to keep these all as separate patches.
I'm happy to squash any number or groups of them. Personally, I find
smaller patches easier to review, and the git blame history acts as
a sort of documentation, so I've kept them for now.

Alexander Graf (1):
  hw/vmapple/apple-gfx: Introduce ParavirtualizedGraphics.Framework
    support

Phil Dennis-Jordan (25):
  hw/vmapple/apple-gfx: BQL renaming update
  hw/display/apple-gfx: Moved from hw/vmapple/
  hw/display/apple-gfx: uses DEFINE_TYPES macro
  hw/display/apple-gfx: native -> little endian memory ops
  hw/display/apple-gfx: Removes dead/superfluous code
  hw/display/apple-gfx: Makes set_mode thread & memory safe
  hw/display/apple-gfx: Adds migration blocker
  hw/display/apple-gfx: Wraps ObjC autorelease code in pool
  hw/display/apple-gfx: Fixes ObjC new/init misuse, plugs leaks
  hw/display/apple-gfx: Uses ObjC category extension for private
    property
  hw/display/apple-gfx: Task memory mapping cleanup
  hw/display/apple-gfx: Defines PGTask_s struct instead of casting
  hw/display/apple-gfx: Refactoring of realize function
  hw/display/apple-gfx: Separates generic & vmapple-specific
    functionality
  hw/display/apple-gfx: Asynchronous MMIO writes on x86-64
  hw/display/apple-gfx: Asynchronous rendering and graphics update
  hw/display/apple-gfx: Adds PCI implementation
  ui/cocoa: Adds non-app runloop on main thread mode
  hw/display/apple-gfx: Fixes cursor hotspot handling
  hw/display/apple-gfx: Implements texture syncing for non-UMA GPUs
  hw/display/apple-gfx: Replaces magic number with queried MMIO length
  hw/display/apple-gfx: Host GPU picking improvements
  hw/display/apple-gfx: Adds configurable mode list
  MAINTAINERS: Add myself as maintainer for apple-gfx, reviewer for HVF
  hw/display/apple-gfx: Removes UI pointer support check

 MAINTAINERS                    |   7 +
 hw/display/Kconfig             |  13 +
 hw/display/apple-gfx-pci.m     | 166 +++++++++
 hw/display/apple-gfx-vmapple.m | 194 ++++++++++
 hw/display/apple-gfx.h         |  72 ++++
 hw/display/apple-gfx.m         | 659 +++++++++++++++++++++++++++++++++
 hw/display/meson.build         |   3 +
 hw/display/trace-events        |  26 ++
 include/qemu-main.h            |   2 +
 meson.build                    |   4 +
 ui/cocoa.m                     |  15 +-
 11 files changed, 1159 insertions(+), 2 deletions(-)
 create mode 100644 hw/display/apple-gfx-pci.m
 create mode 100644 hw/display/apple-gfx-vmapple.m
 create mode 100644 hw/display/apple-gfx.h
 create mode 100644 hw/display/apple-gfx.m

-- 
2.39.3 (Apple Git-146)