Re: [Qemu-devel] Proposal PCI/PCIe device placement on PAPR guests

[Greg Kurz]

Resending because of bad qemu-devel address...

On Thu, 5 Jan 2017 16:46:18 +1100
David Gibson <address@hidden> wrote:

> There was a discussion back in November on the qemu list which spilled
> onto the libvirt list about how to add support for PCIe devices to
> POWER VMs, specifically 'pseries' machine type PAPR guests.
> 
> Here's a more concrete proposal for how to handle part of this in
> future from the libvirt side.  Strictly speaking what I'm suggesting
> here isn't intrinsically linked to PCIe: it will make adding PCIe
> support sanely easier, as well as having a number of advantages for
> both PCIe and plain-PCI devices on PAPR guests.
> 
> Background:
> 
>  * Currently the pseries machine type only supports vanilla PCI
>    buses.
>     * This is a qemu limitation, not something inherent - PAPR guests
>       running under PowerVM (the IBM hypervisor) can use passthrough
>       PCIe devices (PowerVM doesn't emulate devices though).
>     * In fact the way PCI access is para-virtalized in PAPR makes the
>       usual distinctions between PCI and PCIe largely disappear
>  * Presentation of PCIe devices to PAPR guests is unusual
>     * Unlike x86 - and other "bare metal" platforms, root ports are
>       not made visible to the guest. i.e. all devices (typically)
>       appear as though they were integrated devices on x86
>     * In terms of topology all devices will appear in a way similar to
>       a vanilla PCI bus, even PCIe devices
>        * However PCIe extended config space is accessible
>     * This means libvirt's usual placement of PCIe devices is not
>       suitable for PAPR guests
>  * PAPR has its own hotplug mechanism
>     * This is used instead of standard PCIe hotplug
>     * This mechanism works for both PCIe and vanilla-PCI devices
>     * This can hotplug/unplug devices even without a root port P2P
>       bridge between it and the root "bus
>  * Multiple independent host bridges are routine on PAPR
>     * Unlike PC (where all host bridges have multiplexed access to
>       configuration space) PCI host bridges (PHBs) are truly
>       independent for PAPR guests (disjoint MMIO regions in system
>       address space)
>     * PowerVM typically presents a separate PHB to the guest for each
>       host slot passed through
> 
> The Proposal:
> 
> I suggest that libvirt implement a new default algorithm for placing
> (i.e. assigning addresses to) both PCI and PCIe devices for (only)
> PAPR guests.
> 
> The short summary is that by default it should assign each device to a
> separate vPHB, creating vPHBs as necessary.
> 
>   * For passthrough sometimes a group of host devices can't be safely
>     isolated from each other - this is known as a (host) Partitionable
>     Endpoint (PE).  In this case, if any device in the PE is passed
>     through to a guest, the whole PE must be passed through to the
>     same vPHB in the guest.  From the guest POV, each vPHB has exactly
>     one (guest) PE.
>   * To allow for hotplugged devices, libvirt should also add a number
>     of additional, empty vPHBs (the PAPR spec allows for hotplug of
>     PHBs, but this is not yet implemented in qemu).  When hotplugging
>     a new device (or PE) libvirt should locate a vPHB which doesn't
>     currently contain anything.
>   * libvirt should only (automatically) add PHBs - never root ports or
>     other PCI to PCI bridges
> 
> In order to handle migration, the vPHBs will need to be represented in
> the domain XML, which will also allow the user to override this
> topology if they want.
> 
> Advantages:
> 
> There are still some details I need to figure out w.r.t. handling PCIe
> devices (on both the qemu and libvirt sides).  However the fact that

One such detail may be that PCIe devices should have the
"ibm,pci-config-space-type" property set to 1 in the DT,
for the driver to be able to access the extended config
space.

> PAPR guests don't typically see PCIe root ports means that the normal
> libvirt PCIe allocation scheme won't work.  This scheme has several
> advantages with or without support for PCIe devices:
> 
>  * Better performance for 32-bit devices
> 
> With multiple devices on a single vPHB they all must share a (fairly
> small) 32-bit DMA/IOMMU window.  With separate PHBs they each have a
> separate window.  PAPR guests have an always-on guest visible IOMMU.
> 
>  * Better EEH handling for passthrough devices
> 
> EEH is an IBM hardware-assisted mechanism for isolating and safely
> resetting devices experiencing hardware faults so they don't bring
> down other devices or the system at large.  It's roughly similar to
> PCIe AER in concept, but has a different IBM specific interface, and
> works on both PCI and PCIe devices.
> 
> Currently the kernel interfaces for handling EEH events on passthrough
> devices will only work if there is a single (host) iommu group in the
> vfio container.  While lifting that restriction would be nice, it's
> quite difficult to do so (it requires keeping state synchronized
> between multiple host groups).  That also means that an EEH error on
> one device could stop another device where that isn't required by the
> actual hardware.
> 
> The unit of EEH isolation is a PE (Partitionable Endpoint) and
> currently there is only one guest PE per vPHB.  Changing this might
> also be possible, but is again quite complex and may result in
> confusing and/or broken distinctions between groups for EEH isolation
> and IOMMU isolation purposes.
> 
> Placing separate host groups in separate vPHBs sidesteps these
> problems.
> 
>  * Guest NUMA node assignment of devices
> 
> PAPR does not (and can't reasonably) use the pxb device.  Instead to
> allocate devices to different guest NUMA nodes they should be placed
> on different vPHBs.  Placing them on different PHBs by default allows
> NUMA node to be assigned to those PHBs in a straightforward manner.
>

pgpzGluqPgOT_.pgp
Description: OpenPGP digital signature