Re: [Qemu-devel] Secondary VGA passthrough not working, hitting linux kernel's pci_fixup_video

[Michael S. Tsirkin]

On Wed, Jan 15, 2014 at 11:37:29AM +0100, Sander Eikelenboom wrote:
> Hi Michael / Anthony,
> 
> I'm trying to do secondary VGA passthrough with on Xen for some time. The 
> problem was that the driver
> would get the rom of the emulated VGA device instead of it's own rom from the 
> rom bar.
> 
> In the end I found out it is because of a "fixup" in the linux kernel, 
> "pci_fixup_video" in arch/x86/pci/fixup.c
> that sets the IORESOURCE_ROM_SHADOW flag for both VGA cards and make both 
> point to C000 (which is were the emulated rom is).
> 
> The code there tries to verify if the bridge has the PCI_BRIDGE_CTL_VGA bit 
> set, and would only apply the fix when that's true
> however the code doesn't get that far since it returns prematurely since 
> !bridge (the code in question is below as is a dump of lspci -vvvknn).
> 
> The author of that code replied:
> 
>         I understood there is no bridge for a VGA device in your virtual 
> machine.
>         Your emulator for the bridge control register is odd.

That seems beside the point as there's no bridge here.

>         I guess your virtual machine ignore "PCI-to-PCI Bridge Architecture
>         Specification".

Since there's no pci to pci bridge in this VM, why would this
specification apply?

>         Thank you,
> 
>         Eiichiro Oiwa

I can't see that reply anywhere except quoted in your mail.
Tried to hunt addresses for relevant people and Cc them all.
        Cc: address@hidden
        Cc: address@hidden
        Cc: address@hidden
        Cc: address@hidden
        Cc: address@hidden
        Cc: address@hidden

not yet copying until we think it's a linux bug.

> 
> So my first question is .. (i haven't got much knowledge on PCI) is he right 
> and is the layout of pci devices in Qemu indeed out of spec ?
> Or is the pci_fixup_video code assuming too much ... ?
> 
> And second .. any ideas of how to fix it ?
> 
> When i prevent this quirk/fixup from running in the guest kernel, the 
> passthrough works fine .. it can read it's bios .. playing HD video .. even 
> DPM seems to work with the radeon :-)
> 
> Also see this thread on the linux-kernel mailing list: 
> https://lkml.org/lkml/2014/1/11/203


Overall what's the issue here?

Here's my analysis:
You have two video cards both on the root bus.
So the logic in fixup that tries to guess which one
to use based on which one is behind bridge
isn't effective.

As the result fixup will enable ROM shadow for both,
unsurprisingly this doesn't work.

But they can't both work in VGA mode anyway, right?
They would try to claim both addresses.

So I would say you should just disable cirrus on qemu command line with
the -no-graphics flag.
Then it won't conflict.

If you don't want your card to work in VGA mode,
just want it to work in non VGA mode, we run into a problem
in that there's no standard way to disable VGA addresses in devices themselves.
If you want this one solution is to put your card behind a pci to pci bridge in 
qemu.
Our bridges don't support VGA so your card won't work in VGA mode then.

> --
> 
> Sander
> 
> 
> 
> /*
>  * Fixup to mark boot BIOS video selected by BIOS before it changes
>  *
>  * From information provided by "Jon Smirl" <address@hidden>
>  *
>  * The standard boot ROM sequence for an x86 machine uses the BIOS
>  * to select an initial video card for boot display. This boot video
>  * card will have it's BIOS copied to C0000 in system RAM.
>  * IORESOURCE_ROM_SHADOW is used to associate the boot video
>  * card with this copy. On laptops this copy has to be used since
>  * the main ROM may be compressed or combined with another image.
>  * See pci_map_rom() for use of this flag. Before we mark the device
>  * with IORESOURCE_ROM_SHADOW we have to check if this is or should become
>  * the primary video card, since this quirk is ran for all video devices.
>  */
> 
> static void pci_fixup_video(struct pci_dev *pdev)
> {
>         struct pci_dev *bridge;
>         struct pci_bus *bus;
>         u16 config;
> 
>         /* Is VGA routed to us? */
>         bus = pdev->bus;
>         while (bus) {
>                 bridge = bus->self;
> 
>                 /*
>                  * From information provided by
>                  * "David Miller" <address@hidden>
>                  * The bridge control register is valid for PCI header
>                  * type BRIDGE, or CARDBUS. Host to PCI controllers use
>                  * PCI header type NORMAL.
>                  */
>                 if (bridge
>                     && ((bridge->hdr_type == PCI_HEADER_TYPE_BRIDGE)
>                        || (bridge->hdr_type == PCI_HEADER_TYPE_CARDBUS))) {
>                         pci_read_config_word(bridge, PCI_BRIDGE_CONTROL,
>                                                 &config);
>                         if (!(config & PCI_BRIDGE_CTL_VGA))
>                                 return;
>                 }
>                 bus = bus->parent;
>         }
>         if (!vga_default_device() || pdev == vga_default_device()) {
>                 pci_read_config_word(pdev, PCI_COMMAND, &config);
>                 if (config & (PCI_COMMAND_IO | PCI_COMMAND_MEMORY)) {
>                         pdev->resource[PCI_ROM_RESOURCE].flags |= 
> IORESOURCE_ROM_SHADOW;
>                         dev_printk(KERN_DEBUG, &pdev->dev, "Boot video 
> device\n");
>                         vga_set_default_device(pdev);
>                 }
>         }
> }
> DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_ANY_ID, PCI_ANY_ID,
>                                 PCI_CLASS_DISPLAY_VGA, 8, pci_fixup_video);
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> # lspci -vvvknn
> 00:00.0 Host bridge [0600]: Intel Corporation 440FX - 82441FX PMC [Natoma] 
> [8086:1237] (rev 02)
>         Subsystem: Red Hat, Inc Qemu virtual machine [1af4:1100]
>         Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
> Stepping- SERR- FastB2B- DisINTx-
>         Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>         Latency: 0
> 
> 00:01.0 ISA bridge [0601]: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton 
> II] [8086:7000]
>         Subsystem: Red Hat, Inc Qemu virtual machine [1af4:1100]
>         Physical Slot: 1
>         Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
> Stepping- SERR- FastB2B- DisINTx-
>         Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- 
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>         Latency: 0
> 
> 00:01.1 IDE interface [0101]: Intel Corporation 82371SB PIIX3 IDE 
> [Natoma/Triton II] [8086:7010] (prog-if 80 [Master])
>         Subsystem: Red Hat, Inc Qemu virtual machine [1af4:1100]
>         Physical Slot: 1
>         Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
> Stepping- SERR- FastB2B- DisINTx-
>         Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- 
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>         Latency: 0
>         Region 0: [virtual] Memory at 000001f0 (32-bit, non-prefetchable) 
> [size=8]
>         Region 1: [virtual] Memory at 000003f0 (type 3, non-prefetchable) 
> [size=1]
>         Region 2: [virtual] Memory at 00000170 (32-bit, non-prefetchable) 
> [size=8]
>         Region 3: [virtual] Memory at 00000370 (type 3, non-prefetchable) 
> [size=1]
>         Region 4: I/O ports at c260 [size=16]
>         Kernel driver in use: PIIX_IDE
> 
> 00:01.2 USB controller [0c03]: Intel Corporation 82371SB PIIX3 USB 
> [Natoma/Triton II] [8086:7020] (rev 01) (prog-if 00 [UHCI])
>         Subsystem: Red Hat, Inc Qemu virtual machine [1af4:1100]
>         Physical Slot: 1
>         Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
> Stepping- SERR- FastB2B- DisINTx-
>         Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>         Latency: 0
>         Interrupt: pin D routed to IRQ 23
>         Region 4: I/O ports at c240 [size=32]
>         Kernel driver in use: uhci_hcd
> 
> 00:01.3 Bridge [0680]: Intel Corporation 82371AB/EB/MB PIIX4 ACPI [8086:7113] 
> (rev 03)
>         Subsystem: Red Hat, Inc Qemu virtual machine [1af4:1100]
>         Physical Slot: 1
>         Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
> Stepping- SERR- FastB2B- DisINTx-
>         Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- 
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>         Latency: 0
>         Interrupt: pin A routed to IRQ 9
> 
> 00:02.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device 
> [5853:0001] (rev 01)
>         Subsystem: XenSource, Inc. Xen Platform Device [5853:0001]
>         Physical Slot: 2
>         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
> Stepping- SERR- FastB2B- DisINTx-
>         Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>         Latency: 0
>         Interrupt: pin A routed to IRQ 24
>         Region 0: I/O ports at c000 [size=256]
>         Region 1: Memory at f2000000 (32-bit, prefetchable) [size=16M]
>         Kernel driver in use: xen-platform-pci
> 
> 00:03.0 VGA compatible controller [0300]: Cirrus Logic GD 5446 [1013:00b8] 
> (prog-if 00 [VGA controller])
>         Subsystem: Red Hat, Inc Device [1af4:1100]
>         Physical Slot: 3
>         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
> Stepping- SERR- FastB2B- DisINTx-
>         Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>         Latency: 0
>         Region 0: Memory at f0000000 (32-bit, prefetchable) [size=32M]
>         Region 1: Memory at f30b0000 (32-bit, non-prefetchable) [size=4K]
>         Expansion ROM at f30a0000 [disabled] [size=64K]
> 
> 00:05.0 VGA compatible controller [0300]: Advanced Micro Devices [AMD] nee 
> ATI Turks [Radeon HD 6570] [1002:6759] (prog-if 00 [VGA controller])
>         Subsystem: PC Partner Limited Device [174b:e193]
>         Physical Slot: 5
>         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
> Stepping- SERR- FastB2B- DisINTx+
>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>         Latency: 0
>         Interrupt: pin A routed to IRQ 81
>         Region 0: Memory at e0000000 (64-bit, prefetchable) [size=256M]
>         Region 2: Memory at f3060000 (64-bit, non-prefetchable) [size=128K]
>         Region 4: I/O ports at c100 [size=256]
>         Expansion ROM at f3080000 [disabled] [size=128K]
>         Capabilities: [50] Power Management version 3
>                 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA 
> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>                 Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
>         Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00
>                 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, 
> L1 unlimited
>                         ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
>                 DevCtl: Report errors: Correctable- Non-Fatal- Fatal- 
> Unsupported-
>                         RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>                         MaxPayload 128 bytes, MaxReadReq 512 bytes
>                 DevSta: CorrErr+ UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- 
> TransPend-
>                 LnkCap: Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Latency 
> L0 <64ns, L1 <1us
>                         ClockPM- Surprise- LLActRep- BwNot-
>                 LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- 
> CommClk-
>                         ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>                 LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ 
> DLActive- BWMgmt- ABWMgmt-
>                 DevCap2: Completion Timeout: Not Supported, TimeoutDis-
>                 DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
>                 LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- 
> SpeedDis-, Selectable De-emphasis: -6dB
>                          Transmit Margin: Normal Operating Range, 
> EnterModifiedCompliance- ComplianceSOS-
>                          Compliance De-emphasis: -6dB
>                 LnkSta2: Current De-emphasis Level: -3.5dB, 
> EqualizationComplete-, EqualizationPhase1-
>                          EqualizationPhase2-, EqualizationPhase3-, 
> LinkEqualizationRequest-
>         Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
>                 Address: 00000000fee57000  Data: 4300
>         Capabilities: [100 v9] #1002
>         Kernel driver in use: radeon