Re: [PATCH v2 0/9] vfio/igd: Enable legacy mode on more devices

[Tomita Moeko]

On 12/4/24 23:08, Tomita Moeko wrote:
> On 12/4/24 04:12, Alex Williamson wrote:
>> On Tue,  3 Dec 2024 21:35:39 +0800
>> Tomita Moeko <tomitamoeko@gmail.com> wrote:
>>
>>> This patchset extends the support of legacy mode igd passthrough to
>>> all Intel Gen 11 and 12 devices (including Ice Lake, Jasper Lake,
>>> Rocket Lake, Alder Lake and Raptor Lake), and emulates GGC register
>>> in MMIO BAR0 for better compatibiltiy (It is tested Windows and GOP
>>> driver will read this MMIO register).
>>>
>>> It also replaces magic numbers with macros to improve readability,
>>> and aligns behavior (BDSM registor mirroring and GGMS calculation for
>>> gen7) with i915 driver to avoid possible issues.
>>>
>>> The x-igd-gms option removed in 971ca22f041b ("vfio/igd: don't set
>>> stolen memory size to zero") is also added back so that data stolen
>>> memory size can be specified for guest. It is tested that GMS may
>>> related to framebuffer size, a small GMS value may cause display issues
>>> like blackscreen. It can be changed by DVMT Pre-allocated option in
>>> host BIOS, but not all BIOS comes with this option. Having it in QEMU
>>> helps resolves such issues.
>>>
>>> This patchset was verified on Intel i9-12900K CPU(UHD 770, 8086:4680)
>>> with custom OVMF firmware [1] and IntelGopDriver extracted from host
>>> bios. IGD device works well in both Windows and Linux guests, and
>>> scored 726 in 3DMark Time Spy Graphics on Windows guest.
>>>
>>> [1] https://github.com/tomitamoeko/edk2/commits/igd-pt-adl/
>>>
>>> Btw, IO BAR4 seems never be used by guest, and it the IO BAR itself
>>> is not working on Gen11+ devices in my experiments. There is no hints
>>> about that in old commit message and mailing list. It would be greatly
>>> appreciated if someone shares the background.
>>
>> The quirks related to BAR4 access are generally for the vBIOS, we
>> wouldn't expect guest OS level drivers to use them.  IIRC this is
>> handling moving the stolen memory from the HPA to the GPA when the
>> vBIOS is writing the GTT.
> 
> Got it. I'm wondering why vBIOS still writes HPA instead of GPA when
> it's in virtual machine, maybe the address is hardcoded?
> 
>> Maybe that brings up an interesting topic.  Traditionally "legacy mode"
>> IGD assignment has been only for 440fx machines with SeaBIOS and last I
>> was aware edk2 wasn't willing to accept the same hack for the BDSM as
>> we had put into SeaBIOS, instead indicating that it should be
>> implemented in the device ROM.  Your branch in [1] above seems to
>> indicate edk2 does now have assigned IGD specific code.
>>
>> Are these patches developing full stack support of these new devices,
>> from BIOS hand-off, through pre-boot environments, and through to guest
>> OS drivers, or are we only concerned that the guest OS level driver
>> lights up a display?
> 
> Yes these patches provide a complete legacy mode passthrough solution,
> from EFI DXE phase to guest OS, but the EFI part requires specific
> changes in edk2.
> 
>> If you're using q35 and OVMF then you must be operating in the realm of
>> the mythical "Universal Pass-through" mode that I thought Intel had
>> abandoned.  It seems like we need an update to docs/igd-assign.txt as
>> it's likely very out of date based on recent improvements here and by
>> Corvin.
> 
> Actually the only machine supports legacy mode is i440fx, windows driver
> checks the vendor and device id of LPC bridge device at 00:1f.0, if it
> doesn't match, display driver won't work [1]. On q35 machine, there is
> already a emulated ICH9 LPC at 00.1f.0. Previous there was a try in
> modifying the id, but it breaks functionality [2].
> 
> [1] 
> https://github.com/projectacrn/acrn-hypervisor/blob/master/devicemodel/hw/pci/lpc.c#L519
> [2] 
> https://lore.kernel.org/all/1457080913-30018-1-git-send-email-kraxel@redhat.com/
> 
> I had a try "Univerisal Pass-through" mode as igd-assign.txt with my
> UHD 770. On linux guest, it works just as a normal gpu, except there
> is no display output before i915 driver loaded. Even the device's vbdf
> is not 00:02.0, and i915 log shows the DSM is 0M, it works perfectly.
> intel_gpu_top shows igpu is working when I am playing a youtube video.
> 
> I also tried setting primary gpu in bios to discrete gpu, which makes
> the pci class code of igpu changed from 0x030000 (VGA compatible 
> controller) to 0x308000 (Display controller), it can still output to
> monitor connected to it on Linux guest. All with a simple
>     -device vfio-pci,host=00:02.0,id=hostdev0
> 
> But for windows guest, I never had any luck. I attached a virtio-gpu to
> it, with intel grahics drivers installed in guest. If igd is not at
> 00:02.0 or gop driver is not provided, windows BSOD immediately on boot

After further investigation of the i915 kernel driver and additional
testing, I found out the root cause is the VBT inside opregion. VBT
seems to contain configs like how the igd output (Intel call it DVO) is
mapped to physical ports. If VBT is not present, i915 driver is able to
mock one (in 
drivers/gpu/drm/i915/display/intel_bios.c:init_vbt_missing_defaults).
However, windows driver can't, reporting error 43 in device manager or
BSOD. Adding x-igd-opregion=on makes windows guest happy with the UPT
mode. My previous success on linux was just luck, my platform doesn't
has much difference with intel default configuration. OpRegion should
be a must for igd passthrough.

The PCI class code difference is only about whether the device is
chosen as the primary output in BIOS and is mapped to host VGA memory
range or not. The class code change is also observed on intel discrete
GPUs like A750. Both mode can have OpRegion present. Currently kernel
side only probes OpRegion when igd is in VGA class code. I submitted
a patch [1] to enable the probe for display controller mode, both
linux and windows guests can have output to monitor in this mode.  

Probably legacy mode means VGA support at first?

[1] 
https://lore.kernel.org/linux-kernel/20241229155140.7434-1-tomitamoeko@gmail.com/

>> Also, are you proposing the noted edk2 change upstream?  It seems like
>> edk2 would need some sort of device version detection to know whether
>> to use a 32 or 64-bit BDSM value.  Thanks,
>>
>> Alex
> 
> I'm afraid the answer is no, these edk2 changes are not fully open
> source as they were taken from inten directly [3], except the last
> patch. (It seems intel uses a modified qemu as "etc/igd-dsm-base"
> used in patch 5 doesn't exist in qemu, probably they are creating
> a identical GPA->HPA mapping for igd?).
> 
> One of these intel edk2 patches is also included in a edk2 bug [4] and
> markd as hack.
> 
> Besides these edk changes, GOP driver is also needed. Intel never
> released them to public. Even acrn hypervisor developed by intel says
> "Fetch the VBT and GOP drivers from the board manufacturer". The only
> way for users to get it seems to be using tools like "UEFI BIOS
> Updater" to extract driver from host bios image, or try the luck by
> downloading the gop driver shared on internet.
> 
> After having the GOP driver, a virtual rom for igd device in qemu
> can be created by
>     EfiRom -f 0x8086 -i <device_id> -e IgdAssignmentDxe.efi \
>     PlatformGOPPolicy.efi IntelGopDriver.efi
> IgdAssignementDxe.efi and PlatformGOPPolicy.efi are built with the
> edk2 changes.
> 
> I'm not sure whether this can be added to official document.
> 
> [3] 
> https://eci.intel.com/docs/3.0/components/kvm-hypervisor.html#build-ovmf-fd-for-kvm
> [4] https://bugzilla.tianocore.org/show_bug.cgi?id=935
> 
>>> Changelog:
>>> v2:
>>> * Droped "vfio/igd: fix GTT stolen memory size calculation for gen 7".
>>> * Fixed conditions when calculating GGMS size.
>>> * Added Gemini Lake and Comet Lake device ids.
>>> * Splited mirroring register declaration macro into a new patch.
>>> * Minor fixes.
>>> Link: 
>>> 20241201160938.44355-1-tomitamoeko@gmail.com/">https://lore.kernel.org/qemu-devel/20241201160938.44355-1-tomitamoeko@gmail.com/
>>>
>>> Tomita Moeko (9):
>>>   vfio/igd: remove unsupported device ids
>>>   vfio/igd: align generation with i915 kernel driver
>>>   vfio/igd: canonicalize memory size calculations
>>>   vfio/igd: add Gemini Lake and Comet Lake device ids
>>>   vfio/igd: add Alder/Raptor/Rocket/Ice/Jasper Lake device ids
>>>   vfio/igd: add macro for declaring mirrored registers
>>>   vfio/igd: emulate GGC register in mmio bar0
>>>   vfio/igd: emulate BDSM in mmio bar0 for gen 6-10 devices
>>>   vfio/igd: add x-igd-gms option back to set DSM region size for guest
>>>
>>>  hw/vfio/igd.c | 248 ++++++++++++++++++++++++++++++--------------------
>>>  1 file changed, 151 insertions(+), 97 deletions(-)
>>>
>>