Re: [RFC][PATCH] docs: note exception for PCIe IO port access

[Laszlo Ersek]

+Alex

On 06/09/22 04:00, Kevin Locke wrote:
> ioport access is required for VESA BIOS Extensions (VBE).  Since ioport
> access is not forwarded over PCI(e) bridges, graphics adapters must be
> attached directly to the Root Complex in order for the BIOS to provide
> VBE modes.  I'm very grateful to Gerd Hoffmann for explaining this on
> the SeaBIOS mailing list.[1]
>
> Update the PCI Express Guidelines to document this as an exception to
> the recommendation to "Place only legacy PCI devices on the Root
> Complex."
>
> [1]: 
> https://mail.coreboot.org/hyperkitty/list/seabios@seabios.org/thread/XG2RN3HKVRDEDTLA2PRELLIENIIH7II7/#XVP3I2KQVZHSTDA4SNVKOITWGRGSDU3F
>
> Signed-off-by: Kevin Locke <kevin@kevinlocke.name>
> ---

I think we're discussing three separate things here:

(a) plugging a legacy PCI device into an Express slot
(b) the inverse: plugging an Express device into a legacy PCI slot
(c) plugging an Express device into a slot on the PCIe root complex

Telling these apart is not easy because:

> QEMU does not have a clear socket-device matching mechanism
> and allows any PCI/PCI Express device to be plugged into any
> PCI/PCI Express slot.

Now,

(a) your original (non-functional) use case:

qemu-system-x86_64 \
    -no-user-config \
    -nodefaults \
    -machine q35,accel=kvm \
    -m 1G \
    -cdrom "$iso" \
    -device pcie-root-port,id=pci.1,bus=pcie.0 \
    -device VGA,bus=pci.1

violates the following part of "pcie.txt":

> Plugging a PCI device into a PCI Express slot might not always work and
> is weird anyway since it cannot be done for "bare metal".

AIUI, what Gerd explains in that SeaBIOS thread is why and how exactly
such an attempt breaks. The statement that it would break is already
spelled out in "pcie.txt".

So I think that, given strictly your original report on the SeaBIOS
list, no updates to "pcie.txt" are necessary.

Note that your original (functional) use case:

qemu-system-x86_64 \
    -no-user-config \
    -nodefaults \
    -machine q35,accel=kvm \
    -m 1G \
    -cdrom "$iso" \
    -device VGA

does not conflict with

> Place only the following kinds of devices directly on the Root Complex:
>     (1) PCI Devices (e.g. network card, graphics card, IDE controller),
>         not controllers. Place only legacy PCI devices on
>         the Root Complex. These will be considered Integrated Endpoints.
>         Note: Integrated Endpoints are not hot-pluggable.
>
>         Although the PCI Express spec does not forbid PCI Express devices as
>         Integrated Endpoints, existing hardware mostly integrates legacy PCI
>         devices with the Root Complex. Guest OSes are suspected to behave
>         strangely when PCI Express devices are integrated
>         with the Root Complex.

because "-device VGA" is a legacy PCI device, not a PCI Express device.
So the second quoted paragraph does not apply to it at all, and the
first paragraph is in sync with your functional use case.

Then,

On 06/09/22 04:00, Kevin Locke wrote:
> This suggested documentation change is the result of my struggles to
> understand why I was seeing a very limited set of display modes in one
> of my virtual machines, as explained in the seabios ML post linked above
> and an earlier post to qemu-discuss.[2]  I hope it may help avoid some
> of these hassles for future users.

Per your original report, the misconfig on your side was not that you
placed a PCIe display controller on the root complex, but that you
placed a legacy PCI device into a PCIe root port (which is physically
impossible on a physical machine).

>
> I'm far from being an expert in PCI(e), BIOS/VBE, or virtualization in
> general, and would appreciate any suggestions on these docs changes.
> I'm also curious about whether graphics devices are the only exception

While VGA is quirky ("there was only ever intended to be one device" --
see
<http://vfio.blogspot.com/2014/08/whats-deal-with-vga-arbitration.html>),
I maintain that it's sufficiently covered already (although not
specifically) by "pcie.txt"; see above. "-device VGA" is a legacy PCI
device, you can't plug it in an Express slot (root complex or port
alike).

In the blog post at
<https://www.kraxel.org/blog/2019/09/display-devices-in-qemu/>, Gerd
explains that (effectively) only "bochs-display" and "virtio-gpu-pci"
(from QEMU's emulated devices) may be placed in PCI Express slots (root
complex or separate port, alike).

*In theory*, scenario (b) applies to "bochs-display" and
"virtio-gpu-pci" when you plug them into a legacy PCI slot:

> Plugging a PCI Express device into a PCI slot will hide the Extended
> Configuration Space thus is also not recommended.

But this is fine, as these device models don't have extended
capabilities in practice. IOW, I'd imagine them (if they were physical
cards) as PCI Express devices without any use for their extended config
spaces. You could plug them into legacy PCI slots, but would see no
difference in behavior (apart from speed perhaps).

But, we might want to update "pcie.txt" for scenario (c):

On 06/09/22 04:00, Kevin Locke wrote:
> and whether "Guest OSes are suspected to behave strangely when PCI
> Express devices are integrated with the Root Complex" is still the case.

I vaguely recall that Alex didn't fully agree with this statement when
the document was being reviewed.

> David Gibson had previously noted that graphics cards appear on the Root
> Complex on real hardware.[3]

You forgot to provide the link for [3].

Anyway, the full paragraph in the document goes like

>         Although the PCI Express spec does not forbid PCI Express devices as
>         Integrated Endpoints, existing hardware mostly integrates legacy PCI
>         devices with the Root Complex. Guest OSes are suspected to behave
>         strangely when PCI Express devices are integrated
>         with the Root Complex.

So the recommendation is based on motherboards seen in the wild, and the
potential consequent assumptions in OSes. Whether that still holds up
with today's motherboards (and OSes), I have no clue about.

>
> I notice that the HD Audio Controller on my T430 is an "Express Root
> Complex Integrated Endpoint".  Is this a non-graphics example, or am I
> misinterpreting the lspci output?
>
> 00:1b.0 Audio device: Intel Corporation 7 Series/C216 Chipset Family High 
> Definition Audio Controller (rev 04)
>       Subsystem: Lenovo 7 Series/C216 Chipset Family High Definition Audio 
> Controller
>       Flags: bus master, fast devsel, latency 0, IRQ 29, IOMMU group 6
>       Memory at f1530000 (64-bit, non-prefetchable) [size=16K]
>       Capabilities: [50] Power Management version 2
>       Capabilities: [60] MSI: Enable+ Count=1/1 Maskable- 64bit+
>       Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00
>       Capabilities: [100] Virtual Channel
>       Capabilities: [130] Root Complex Link
>       Kernel driver in use: snd_hda_intel
>       Kernel modules: snd_hda_intel
>
> Thanks,
> Kevin
>
>  docs/pcie.txt | 5 +++++
>  1 file changed, 5 insertions(+)
>
> diff --git a/docs/pcie.txt b/docs/pcie.txt
> index 89e3502075..a23d93849b 100644
> --- a/docs/pcie.txt
> +++ b/docs/pcie.txt
> @@ -48,6 +48,11 @@ Place only the following kinds of devices directly on the 
> Root Complex:
>          strangely when PCI Express devices are integrated
>          with the Root Complex.
>
> +        An exception to this rule is PCI Express devices which will be
> +        accessed using IO ports.

I find this too general; a PCI Express device is supposed to work
without IO resources. Graphics cards with legacy VGA compatibility are
the exception AIUI (see again Alex's blog about VGA arbitration), so we
should spell that out.

Furthermore, from QEMU's emulated graphics cards, all device models that
provide VGA compatibility are legacy PCI (not PCI Express) devices (see
again Gerd's post: VGA, virtio-vga, qxl-vga, cirrus-vga, ati-vga), so
I'd even restrict this docs update to assigned GPUs.

>                                   For example, guests using BIOS firmware
> +        require IO port access for graphics devices to provide VESA BIOS
> +        Extensions (VBE).
> +
>      (2) PCI Express Root Ports (ioh3420), for starting exclusively PCI 
> Express
>          hierarchies.
>
>

How about inserting a new paragraph here, instead:

    (2) Assigned PCI Express GPUs that offer legacy VGA compatibility,
        and that such compatibility is expected of (due to booting with
        SeaBIOS, or due to UEFI driver bugs or native OS driver bugs).

Thanks
Laszlo