Re: [Qemu-devel] Attaching PCI devices to the PCIe root complex

[Markus Armbruster]

Marcel Apfelbaum <address@hidden> writes:

> On Wed, 2013-09-25 at 10:01 +0300, Michael S. Tsirkin wrote:
>> On Tue, Sep 24, 2013 at 06:01:02AM -0400, Laine Stump wrote:
>> > When I added support for the Q35-based machinetypes to libvirt, I
>> > specifically prohibited attaching any PCI devices (with the exception of
>> > graphics controllers) to the PCIe root complex,
>> 
>> That's wrong I think. Anything attached to RC is an integrated
>> endpoint, and these can be PCI devices.
> I couldn't find on PCIe spec any mention that "Root Complex Integrated 
> EndPoint"
> must be PCIe. But, from spec 1.3.2.3:
> - A Root Complex Integrated Endpoint must not require I/O resources
> claimed through BAR(s).
> - A Root Complex Integrated Endpoint must not generate I/O Requests.
> - A Root Complex Integrated Endpoint is required to support MSI or
> MSI-X or both if an
> interrupt resource is requested.
> I suppose that this restriction can be removed for PCI devices that
> 1. Actually work when plugged in into RC Integrated EndPoint
> 2. Respond to the above limitations
>
>> 
>> > and had planned to
>> > prevent attaching them to PCIe root ports (ioh3420 device) and PCIe
>> > downstream switch ports (xio-3130 device) as well. I did this because,
>> > even though qemu currently allows attaching a normal PCI device in any
>> > of these three places, the restriction exists for real hardware and I
>> > didn't see any guarantee that qemu wouldn't add the restriction in the
>> > future in order to more closely emulate real hardware.
>> > 
>> > However, since I did that, I've learned that many of the qemu "pci"
>> > devices really should be considered as "pci or pcie". Gerd Hoffman lists
>> > some of these cases in a bug he filed against libvirt:
>> > 
>> >    https://bugzilla.redhat.com/show_bug.cgi?id=1003983
>> > 
>> > I would like to loosen up the restrictions in libvirt, but want to make
>> > sure that I don't allow something that could later be forbidden by qemu
>> > (thus creating a compatibility problem during upgrades). Beyond Gerd's
>> > specific requests to allow ehci, uhci, and hda controllers to attach to
>> > PCIe ports, are there any other devices that I specifically should or
>> > shouldn't allow? (I would rather be conservative in what I allow - it's
>> > easy to allow more things later, but nearly impossible to revoke
>> > permission once it's been allowed).
> For the moment I would not remove any restrictions, but only the ones
> requested and verified by somebody.
>
>> 
>> IMO, we really need to grow an interface to query this kind of thing.
> Basically libvirt needs to know:
> 1. for (libvirt) controllers: what kind of devices can be plugged in
> 2. for devices (controller is also a device)
>     - to which controllers can it be plugged in
>     - does it support hot-plug?
> 3. implicit controllers of the machine types (q35 - "pcie-root", i440fx - 
> "pci-root")
> All the above must be exported to libvirt
>
> Implementation options:
> 1. Add a compliance field on PCI/PCIe devices and controllers stating if it 
> supports
>    PCI/PCIe or both (and maybe hot-plug)
>    - consider plug type + compliance to figure out whether a plug can go into 
> a socket
>    
> 2. Use Markus Armbruster idea of introducing a concept of "plug and sockets":
>    - dividing the devices into adapters and plugs
>    - adding sockets to bridges(buses?).
>    In this way it would be clear which devices can connect to bridges

This isn't actually my idea.  It's how things are designed to work in
qdev, at least in my admittedly limited understanding of qdev.

In traditional qdev, a device has exactly one plug (its "bus type",
shown by -device help), and it may have one ore more buses.  Each bus
has a type, and you can plug a device only into a bus of the matching
type.  This was too limiting, and is not how things work now.

As far as I know, libvirt already understands that a device can only
plug into a matching bus.

In my understanding of where we're headed with qdev, things are / will
become more general, yet stay really simple:

* A device can have an arbitrary number of sockets and plugs.

* Each socket / plug has a type.

* A plug can go into a socket of the same type, and only there.

Pretty straightforward generalization of traditional qdev.  I wouldn't
expect libvirt to have serious trouble coping with it, as long as we
provide the necessary information on device models' plugs and sockets.

In this framework, there's no such thing as a device model that can plug
either into a PCI or a PCIe socket.  Makes sense to me, because there's
no such thing in the physical world, either.

Instead, and just like in the physical world, you have one separate
device variant per desired plug type.

To get that, you have to split the device into a common core and bus
adapters.  You compose the core with the PCI adapter to get the PCI
device, with the PCIe adapter to get the PCIe device, and so forth.

I'm not claiming that's the best way to do PCI + PCIe.  It's a purely
theoretical approach, concerned only with conceptual cleanliness, not
practical coding difficulties.

What we have now is entirely different: we've overloaded the existing
PCI plug with all the other PCI-related plug types that came with the
PCIe support, so it means pretty much nothing anymore.  In particular,
there's no way for libvirt to figure out progragmatically whether some
alleged "PCI" device can go into some alleged "PCI" bus.  I call that a
mess.