Re: [PATCH v2 0/9] vfio/igd: Enable legacy mode on more devices

[Tomita Moeko]

On 12/4/24 04:12, Alex Williamson wrote:
> On Tue,  3 Dec 2024 21:35:39 +0800
> Tomita Moeko <tomitamoeko@gmail.com> wrote:
> 
>> This patchset extends the support of legacy mode igd passthrough to
>> all Intel Gen 11 and 12 devices (including Ice Lake, Jasper Lake,
>> Rocket Lake, Alder Lake and Raptor Lake), and emulates GGC register
>> in MMIO BAR0 for better compatibiltiy (It is tested Windows and GOP
>> driver will read this MMIO register).
>>
>> It also replaces magic numbers with macros to improve readability,
>> and aligns behavior (BDSM registor mirroring and GGMS calculation for
>> gen7) with i915 driver to avoid possible issues.
>>
>> The x-igd-gms option removed in 971ca22f041b ("vfio/igd: don't set
>> stolen memory size to zero") is also added back so that data stolen
>> memory size can be specified for guest. It is tested that GMS may
>> related to framebuffer size, a small GMS value may cause display issues
>> like blackscreen. It can be changed by DVMT Pre-allocated option in
>> host BIOS, but not all BIOS comes with this option. Having it in QEMU
>> helps resolves such issues.
>>
>> This patchset was verified on Intel i9-12900K CPU(UHD 770, 8086:4680)
>> with custom OVMF firmware [1] and IntelGopDriver extracted from host
>> bios. IGD device works well in both Windows and Linux guests, and
>> scored 726 in 3DMark Time Spy Graphics on Windows guest.
>>
>> [1] https://github.com/tomitamoeko/edk2/commits/igd-pt-adl/
>>
>> Btw, IO BAR4 seems never be used by guest, and it the IO BAR itself
>> is not working on Gen11+ devices in my experiments. There is no hints
>> about that in old commit message and mailing list. It would be greatly
>> appreciated if someone shares the background.
> 
> The quirks related to BAR4 access are generally for the vBIOS, we
> wouldn't expect guest OS level drivers to use them.  IIRC this is
> handling moving the stolen memory from the HPA to the GPA when the
> vBIOS is writing the GTT.

Got it. I'm wondering why vBIOS still writes HPA instead of GPA when
it's in virtual machine, maybe the address is hardcoded?

> Maybe that brings up an interesting topic.  Traditionally "legacy mode"
> IGD assignment has been only for 440fx machines with SeaBIOS and last I
> was aware edk2 wasn't willing to accept the same hack for the BDSM as
> we had put into SeaBIOS, instead indicating that it should be
> implemented in the device ROM.  Your branch in [1] above seems to
> indicate edk2 does now have assigned IGD specific code.
>
> Are these patches developing full stack support of these new devices,
> from BIOS hand-off, through pre-boot environments, and through to guest
> OS drivers, or are we only concerned that the guest OS level driver
> lights up a display?

Yes these patches provide a complete legacy mode passthrough solution,
from EFI DXE phase to guest OS, but the EFI part requires specific
changes in edk2.

> If you're using q35 and OVMF then you must be operating in the realm of
> the mythical "Universal Pass-through" mode that I thought Intel had
> abandoned.  It seems like we need an update to docs/igd-assign.txt as
> it's likely very out of date based on recent improvements here and by
> Corvin.

Actually the only machine supports legacy mode is i440fx, windows driver
checks the vendor and device id of LPC bridge device at 00:1f.0, if it
doesn't match, display driver won't work [1]. On q35 machine, there is
already a emulated ICH9 LPC at 00.1f.0. Previous there was a try in
modifying the id, but it breaks functionality [2].

[1] 
https://github.com/projectacrn/acrn-hypervisor/blob/master/devicemodel/hw/pci/lpc.c#L519
[2] 
https://lore.kernel.org/all/1457080913-30018-1-git-send-email-kraxel@redhat.com/

I had a try "Univerisal Pass-through" mode as igd-assign.txt with my
UHD 770. On linux guest, it works just as a normal gpu, except there
is no display output before i915 driver loaded. Even the device's vbdf
is not 00:02.0, and i915 log shows the DSM is 0M, it works perfectly.
intel_gpu_top shows igpu is working when I am playing a youtube video.

I also tried setting primary gpu in bios to discrete gpu, which makes
the pci class code of igpu changed from 0x030000 (VGA compatible 
controller) to 0x308000 (Display controller), it can still output to
monitor connected to it on Linux guest. All with a simple
    -device vfio-pci,host=00:02.0,id=hostdev0

But for windows guest, I never had any luck. I attached a virtio-gpu to
it, with intel grahics drivers installed in guest. If igd is not at
00:02.0 or gop driver is not provided, windows BSOD immediately on boot

> Also, are you proposing the noted edk2 change upstream?  It seems like
> edk2 would need some sort of device version detection to know whether
> to use a 32 or 64-bit BDSM value.  Thanks,
> 
> Alex

I'm afraid the answer is no, these edk2 changes are not fully open
source as they were taken from inten directly [3], except the last
patch. (It seems intel uses a modified qemu as "etc/igd-dsm-base"
used in patch 5 doesn't exist in qemu, probably they are creating
a identical GPA->HPA mapping for igd?).

One of these intel edk2 patches is also included in a edk2 bug [4] and
markd as hack.

Besides these edk changes, GOP driver is also needed. Intel never
released them to public. Even acrn hypervisor developed by intel says
"Fetch the VBT and GOP drivers from the board manufacturer". The only
way for users to get it seems to be using tools like "UEFI BIOS
Updater" to extract driver from host bios image, or try the luck by
downloading the gop driver shared on internet.

After having the GOP driver, a virtual rom for igd device in qemu
can be created by
    EfiRom -f 0x8086 -i <device_id> -e IgdAssignmentDxe.efi \
    PlatformGOPPolicy.efi IntelGopDriver.efi
IgdAssignementDxe.efi and PlatformGOPPolicy.efi are built with the
edk2 changes.

I'm not sure whether this can be added to official document.

[3] 
https://eci.intel.com/docs/3.0/components/kvm-hypervisor.html#build-ovmf-fd-for-kvm
[4] https://bugzilla.tianocore.org/show_bug.cgi?id=935

>> Changelog:
>> v2:
>> * Droped "vfio/igd: fix GTT stolen memory size calculation for gen 7".
>> * Fixed conditions when calculating GGMS size.
>> * Added Gemini Lake and Comet Lake device ids.
>> * Splited mirroring register declaration macro into a new patch.
>> * Minor fixes.
>> Link: 
>> 20241201160938.44355-1-tomitamoeko@gmail.com/">https://lore.kernel.org/qemu-devel/20241201160938.44355-1-tomitamoeko@gmail.com/
>>
>> Tomita Moeko (9):
>>   vfio/igd: remove unsupported device ids
>>   vfio/igd: align generation with i915 kernel driver
>>   vfio/igd: canonicalize memory size calculations
>>   vfio/igd: add Gemini Lake and Comet Lake device ids
>>   vfio/igd: add Alder/Raptor/Rocket/Ice/Jasper Lake device ids
>>   vfio/igd: add macro for declaring mirrored registers
>>   vfio/igd: emulate GGC register in mmio bar0
>>   vfio/igd: emulate BDSM in mmio bar0 for gen 6-10 devices
>>   vfio/igd: add x-igd-gms option back to set DSM region size for guest
>>
>>  hw/vfio/igd.c | 248 ++++++++++++++++++++++++++++++--------------------
>>  1 file changed, 151 insertions(+), 97 deletions(-)
>>
>