[Qemu-devel] VFIO PCIe Extended Capabilities

[Spenser Gilliland]

Hi,

I noticed your patches "vfio: add pcie extended capability support" and 
"vfio/pci: Hide SR-IOV capability" have gone into qemu mainline.  These look 
really good, and thanks so much for doing these.

I was wondering if there were any side effects to removing the 
pci_bus_is_express check on line 1776 of hw/vfio/pci.c .

    /* Only add extended caps if we have them and the guest can see them */
-   if (!pci_is_express(pdev) || !pci_bus_is_express(pdev->bus) ||
+  if (!pci_is_express(pdev) ||
        !pci_get_long(pdev->config + PCI_CONFIG_SPACE_SIZE)) {
        return 0;
    }

I'm asking because it looks like the defaults for libvirt/OpenStack are to 
create a "hostdev" stanza in the libvirt xml to define this pass through 
condition.  However, it also appears that the "hostdev" stanza only supports 
pci-bridge bus connections.  Thus, it's not easily possible to use this patch 
in a libvirt/OpenStack environment as the bus is technically a non-express bus. 
 It looks like adding PCIe bus support to libvirt/OpenStack may be a lot more 
effort than a simple workaround here.

I have tested this on my local system and it does work as intended for my use 
case.  The following is from an OpenStack VM and shows that the 0x340 extended 
configuration space is passed through correctly.  I've also done testing which 
uses this space and the results are positive.

# lspci -tvnn
-[0000:00]-+-00.0  Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller 
[8086:29c0]
           +-01.0  Cirrus Logic GD 5446 [1013:00b8]
           +-1e.0-[01-02]----01.0-[02]--+-01.0  Red Hat, Inc Virtio network 
device [1af4:1000]
           |                            +-02.0  Red Hat, Inc Virtio block 
device [1af4:1001]
           |                            +-03.0  Red Hat, Inc Virtio block 
device [1af4:1001]
           |                            +-04.0  Xilinx Corporation Device 
[10ee:8138]
           |                            \-05.0  Red Hat, Inc Virtio memory 
balloon [1af4:1002]
           +-1f.0  Intel Corporation 82801IB (ICH9) LPC Interface Controller 
[8086:2918]
           +-1f.2  Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA 
Controller [AHCI mode] [8086:2922]
           \-1f.3  Intel Corporation 82801I (ICH9 Family) SMBus Controller 
[8086:2930]

# lspci -vvv -d 10ee:
02:04.0 Memory controller: Xilinx Corporation Device 8138
        Subsystem: Xilinx Corporation Device 0121
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- 
<MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 29
        Region 0: Memory at fdc00000 (32-bit, non-prefetchable) [size=4M]
        Region 1: Memory at fe000000 (32-bit, non-prefetchable) [size=1M]
        Capabilities: [80] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA 
PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit+
                Address: 00000000fee00000  Data: 4099
        Capabilities: [c0] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, 
L1 <1us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ 
Unsupported+
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq- AuxPwr- 
TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM unknown, Latency 
L0 unlimited, L1 unlimited
                        ClockPM- Surprise- LLActRep- BwNot-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- 
BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range B, TimeoutDis+, LTR-, OBFF 
Not Supported
                DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-, LTR-, 
OBFF Disabled
                LnkSta2: Current De-emphasis Level: -3.5dB, 
EqualizationComplete+, EqualizationPhase1+
                         EqualizationPhase2+, EqualizationPhase3+, 
LinkEqualizationRequest-
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- 
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt+ 
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP+ FCP+ CmpltTO+ CmpltAbrt- UnxCmplt- 
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr+ BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                CEMsk:  RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
        Capabilities: [340 v1] Vendor Specific Information: ID=0001 Rev=0 
Len=02c <?>
        Kernel driver in use: xcldma
        Kernel modules: xcldma

Thanks,
Spenser


This email and any attachments are intended for the sole use of the named 
recipient(s) and contain(s) confidential information that may be proprietary, 
privileged or copyrighted under applicable law. If you are not the intended 
recipient, do not read, copy, or forward this email message or any attachments. 
Delete this email message and any attachments immediately.