Re: [Qemu-devel] Fwd: [RFC PATCH 0/2] Allow RedHat PCI bridges reserve more buses than necessary during init

[Marcel Apfelbaum]

On 19/07/2017 21:56, Konrad Rzeszutek Wilk wrote:
> On Wed, Jul 19, 2017 at 09:38:50PM +0300, Alexander Bezzubikov wrote:
> > 2017-07-19 21:18 GMT+03:00 Konrad Rzeszutek Wilk <address@hidden>:
> >
> > > On Wed, Jul 19, 2017 at 05:14:41PM +0000, Alexander Bezzubikov wrote:
> > > > ср, 19 июля 2017 г. в 16:57, Konrad Rzeszutek Wilk <
> > > address@hidden>:
> > > > > On Wed, Jul 19, 2017 at 04:20:12PM +0300, Aleksandr Bezzubikov wrote:
> > > > > > Now PCI bridges (and PCIE root port too) get a bus range number in
> > > > > system init,
> > > > > > basing on currently plugged devices. That's why when one wants to
> > > > > hotplug another bridge,
> > > > > > it needs his child bus, which the parent is unable to provide.
> > > > >
> > > > > Could you explain how you trigger this?
> > > >
> > > >
> > > > I'm trying to hot plug pcie-pci bridge into pcie root port, and Linux
> > > says
> > > > 'cannot allocate bus number for device bla-bla'. This obviously does not
> > > > allow me to use the bridge at all.
> > > >
> > > > > > The suggested workaround is to have vendor-specific capability in
> > > RedHat
> > > > > generic pcie-root-port
> > > > > > that contains number of additional bus to reserve on BIOS PCI init.
> > > > >
> > > > > But wouldn't the proper fix be for the PCI bridge to have the
> > > subordinate
> > > > > value be extended to fit more bus ranges?
> > > >
> > > >
> > > > What do you mean? This is what I'm trying to do. Do you suppose to get
> > > rid
> > > > of vendor-specific cap and use original register value instead of it?
> > >
> > > I would suggest a simple fix - each bridge has a a number of bus devices
> > > it can use. You have up to 255 - so you split the number of northbridge
> > > numbers by the amount of NUMA nodes (if that is used) - so for example
> > > if you have 4 NUMA nodes, each bridge would cover 63 bus numbers.
> > >
> > > Meaning the root bridge would cover 0->63 bus, 64->128, and so on.
> > > That gives you enough space to plug in your plugged in devices
> > > (up to 63).
> > >
> > > And if you need sub-briges then carve out a specific range.

Hi Konrad,

> > The problem is that we don't know at the init moment how many subbridges we
> > may need,

Is possible the explanation was not clear clear and led to
some miscommunication.

>                                                                                                            
> And the explanation above does not either. It just setups at init time
> an range where you can plug in your new devices in. But in a more uniform
> way such that you can also utilize this with NUMA and _PXM topology
> in the future.

I fully agree with you and actually QEMU has already implemented the
exact idea you are describing here, its called a pxb/pxb-pci device,
that can be "bounded" to a specific NUMA node and has a subrange of bus
numbers dedicated to it.

However this problem is different. In a PCI Express machine you
can hotplug PCIe devices only into PCIe Root Ports (or switch
downstream ports, but not in current scope).

We want to be able to hotplug a PCIe-PCI bridge into a PCIe Root Port
so we can then hot-plug legacy PCI devices.

Since the PCIe Root Port is a type of PCI bridge, at boot time
it only gets the bus sub-range (primary bus,subordinate bus]
which is computed by firmware and leaves no bus number that
can be used by a hot-plugged pci-bridge. And this obviously
does not depend on how we arrange NUMA/proximities.

We are also not looking for a fix for a specific guest OS,
so reserving some extra bus-numbers it has minimal impact
on the system. I do agree the problem may be solved differently,
however we can't reach all guest OS vendors and ask them to
support an alternative solution in a reasonable time frame.

Thanks,
Marcel

> > and how deep the whole device tree will be. The key moment - PCI bridge
> > hotplugging
> > needs either rescan all buses on each bridge device addition, or reserve
> > space in advance during BIOS init.
>
> can all buses on each bridge device addition, or reserve
>                                                                                                                                      
> It is more complex than that - you may need to move devices that are
> below you. And Linux kernel (nor any other OS) can handle that.
> (They can during bootup)
>
> > In this series the second way was chosen.
> >
> >
> > > > > > Aleksandr Bezzubikov (2):
> > > > > >    pci: add support for direct usage of bdf for capability lookup
> > > > > >    pci: enable RedHat pci bridges to reserve more buses
> > > > > >
> > > > > >   src/fw/pciinit.c   | 12 ++++++++++--
> > > > > >   src/hw/pcidevice.c | 24 ++++++++++++++++++++++++
> > > > > >   src/hw/pcidevice.h |  1 +
> > > > > >   3 files changed, 35 insertions(+), 2 deletions(-)
> > > > > >
> > > > > > --
> > > > > > 2.7.4
> > > > --
> > > > Alexander Bezzubikov
> >
> >
> >
> > --
> > Alexander Bezzubikov