[Qemu-devel] Secondary VGA passthrough not working, hitting linux kernel's pci_fixup_video

[Sander Eikelenboom]

Hi Michael / Anthony,

I'm trying to do secondary VGA passthrough with on Xen for some time. The 
problem was that the driver
would get the rom of the emulated VGA device instead of it's own rom from the 
rom bar.

In the end I found out it is because of a "fixup" in the linux kernel, 
"pci_fixup_video" in arch/x86/pci/fixup.c
that sets the IORESOURCE_ROM_SHADOW flag for both VGA cards and make both point 
to C000 (which is were the emulated rom is).

The code there tries to verify if the bridge has the PCI_BRIDGE_CTL_VGA bit 
set, and would only apply the fix when that's true
however the code doesn't get that far since it returns prematurely since 
!bridge (the code in question is below as is a dump of lspci -vvvknn).

The author of that code replied:

        I understood there is no bridge for a VGA device in your virtual 
machine.
        Your emulator for the bridge control register is odd.
        I guess your virtual machine ignore "PCI-to-PCI Bridge Architecture
        Specification".

        Thank you,

        Eiichiro Oiwa


So my first question is .. (i haven't got much knowledge on PCI) is he right 
and is the layout of pci devices in Qemu indeed out of spec ?
Or is the pci_fixup_video code assuming too much ... ?

And second .. any ideas of how to fix it ?

When i prevent this quirk/fixup from running in the guest kernel, the 
passthrough works fine .. it can read it's bios .. playing HD video .. even DPM 
seems to work with the radeon :-)

Also see this thread on the linux-kernel mailing list: 
https://lkml.org/lkml/2014/1/11/203

--

Sander



/*
 * Fixup to mark boot BIOS video selected by BIOS before it changes
 *
 * From information provided by "Jon Smirl" <address@hidden>
 *
 * The standard boot ROM sequence for an x86 machine uses the BIOS
 * to select an initial video card for boot display. This boot video
 * card will have it's BIOS copied to C0000 in system RAM.
 * IORESOURCE_ROM_SHADOW is used to associate the boot video
 * card with this copy. On laptops this copy has to be used since
 * the main ROM may be compressed or combined with another image.
 * See pci_map_rom() for use of this flag. Before we mark the device
 * with IORESOURCE_ROM_SHADOW we have to check if this is or should become
 * the primary video card, since this quirk is ran for all video devices.
 */

static void pci_fixup_video(struct pci_dev *pdev)
{
        struct pci_dev *bridge;
        struct pci_bus *bus;
        u16 config;

        /* Is VGA routed to us? */
        bus = pdev->bus;
        while (bus) {
                bridge = bus->self;

                /*
                 * From information provided by
                 * "David Miller" <address@hidden>
                 * The bridge control register is valid for PCI header
                 * type BRIDGE, or CARDBUS. Host to PCI controllers use
                 * PCI header type NORMAL.
                 */
                if (bridge
                    && ((bridge->hdr_type == PCI_HEADER_TYPE_BRIDGE)
                       || (bridge->hdr_type == PCI_HEADER_TYPE_CARDBUS))) {
                        pci_read_config_word(bridge, PCI_BRIDGE_CONTROL,
                                                &config);
                        if (!(config & PCI_BRIDGE_CTL_VGA))
                                return;
                }
                bus = bus->parent;
        }
        if (!vga_default_device() || pdev == vga_default_device()) {
                pci_read_config_word(pdev, PCI_COMMAND, &config);
                if (config & (PCI_COMMAND_IO | PCI_COMMAND_MEMORY)) {
                        pdev->resource[PCI_ROM_RESOURCE].flags |= 
IORESOURCE_ROM_SHADOW;
                        dev_printk(KERN_DEBUG, &pdev->dev, "Boot video 
device\n");
                        vga_set_default_device(pdev);
                }
        }
}
DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_ANY_ID, PCI_ANY_ID,
                                PCI_CLASS_DISPLAY_VGA, 8, pci_fixup_video);













# lspci -vvvknn
00:00.0 Host bridge [0600]: Intel Corporation 440FX - 82441FX PMC [Natoma] 
[8086:1237] (rev 02)
        Subsystem: Red Hat, Inc Qemu virtual machine [1af4:1100]
        Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- 
<MAbort- >SERR- <PERR- INTx-
        Latency: 0

00:01.0 ISA bridge [0601]: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton 
II] [8086:7000]
        Subsystem: Red Hat, Inc Qemu virtual machine [1af4:1100]
        Physical Slot: 1
        Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- 
<TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0

00:01.1 IDE interface [0101]: Intel Corporation 82371SB PIIX3 IDE 
[Natoma/Triton II] [8086:7010] (prog-if 80 [Master])
        Subsystem: Red Hat, Inc Qemu virtual machine [1af4:1100]
        Physical Slot: 1
        Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- 
<TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Region 0: [virtual] Memory at 000001f0 (32-bit, non-prefetchable) 
[size=8]
        Region 1: [virtual] Memory at 000003f0 (type 3, non-prefetchable) 
[size=1]
        Region 2: [virtual] Memory at 00000170 (32-bit, non-prefetchable) 
[size=8]
        Region 3: [virtual] Memory at 00000370 (type 3, non-prefetchable) 
[size=1]
        Region 4: I/O ports at c260 [size=16]
        Kernel driver in use: PIIX_IDE

00:01.2 USB controller [0c03]: Intel Corporation 82371SB PIIX3 USB 
[Natoma/Triton II] [8086:7020] (rev 01) (prog-if 00 [UHCI])
        Subsystem: Red Hat, Inc Qemu virtual machine [1af4:1100]
        Physical Slot: 1
        Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- 
<MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin D routed to IRQ 23
        Region 4: I/O ports at c240 [size=32]
        Kernel driver in use: uhci_hcd

00:01.3 Bridge [0680]: Intel Corporation 82371AB/EB/MB PIIX4 ACPI [8086:7113] 
(rev 03)
        Subsystem: Red Hat, Inc Qemu virtual machine [1af4:1100]
        Physical Slot: 1
        Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- 
<TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 9

00:02.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device 
[5853:0001] (rev 01)
        Subsystem: XenSource, Inc. Xen Platform Device [5853:0001]
        Physical Slot: 2
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- 
<MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 24
        Region 0: I/O ports at c000 [size=256]
        Region 1: Memory at f2000000 (32-bit, prefetchable) [size=16M]
        Kernel driver in use: xen-platform-pci

00:03.0 VGA compatible controller [0300]: Cirrus Logic GD 5446 [1013:00b8] 
(prog-if 00 [VGA controller])
        Subsystem: Red Hat, Inc Device [1af4:1100]
        Physical Slot: 3
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
Stepping- SERR- FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- 
<MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Region 0: Memory at f0000000 (32-bit, prefetchable) [size=32M]
        Region 1: Memory at f30b0000 (32-bit, non-prefetchable) [size=4K]
        Expansion ROM at f30a0000 [disabled] [size=64K]

00:05.0 VGA compatible controller [0300]: Advanced Micro Devices [AMD] nee ATI 
Turks [Radeon HD 6570] [1002:6759] (prog-if 00 [VGA controller])
        Subsystem: PC Partner Limited Device [174b:e193]
        Physical Slot: 5
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- 
<MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 81
        Region 0: Memory at e0000000 (64-bit, prefetchable) [size=256M]
        Region 2: Memory at f3060000 (64-bit, non-prefetchable) [size=128K]
        Region 4: I/O ports at c100 [size=256]
        Expansion ROM at f3080000 [disabled] [size=128K]
        Capabilities: [50] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA 
PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 
unlimited
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- 
Unsupported-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- 
TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Latency 
L0 <64ns, L1 <1us
                        ClockPM- Surprise- LLActRep- BwNot-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ 
DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, 
Selectable De-emphasis: -6dB
                         Transmit Margin: Normal Operating Range, 
EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, 
EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, 
LinkEqualizationRequest-
        Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
                Address: 00000000fee57000  Data: 4300
        Capabilities: [100 v9] #1002
        Kernel driver in use: radeon