Re: [Qemu-ppc] [Qemu-devel] [PATCH v5 for-2.3 28/28] docs: Add PXB documentation

[Marcel Apfelbaum]

On 03/16/2015 05:28 PM, Michael S. Tsirkin wrote:
> On Mon, Mar 16, 2015 at 02:16:40PM +0200, Marcel Apfelbaum wrote:
> > On 03/10/2015 07:42 PM, Michael S. Tsirkin wrote:
> > > On Tue, Mar 10, 2015 at 06:21:14PM +0200, Marcel Apfelbaum wrote:
> > > > On 03/10/2015 05:47 PM, Michael S. Tsirkin wrote:
> > > > > On Tue, Mar 10, 2015 at 05:32:14PM +0200, Marcel Apfelbaum wrote:
> > > > > > Signed-off-by: Marcel Apfelbaum <address@hidden>
> > > > > > ---
> > > > > >   docs/pci_expander_bridge.txt | 52 ++++++++++++++++++++++++++++++++++++++++++++
> > > > > >   1 file changed, 52 insertions(+)
> > > > > >   create mode 100644 docs/pci_expander_bridge.txt
> > > > > >
> > > > > > diff --git a/docs/pci_expander_bridge.txt b/docs/pci_expander_bridge.txt
> > > > > > new file mode 100644
> > > > > > index 0000000..58bf7a8
> > > > > > --- /dev/null
> > > > > > +++ b/docs/pci_expander_bridge.txt
> > > > > > @@ -0,0 +1,52 @@
> > > > > > +PCI EXPANDER BRIDGE (PXB)
> > > > > > +=========================
> > > > > > +
> > > > > > +Description
> > > > > > +===========
> > > > > > +PXB is a "light-weight" host bridge in the same PCI domain
> > > > > > +as the main host bridge whose purpose is to enable
> > > > > > +the main host bridge to support multiple PCI root buses.
> > > > > > +It is implemented only for i440fx.
> > > > >
> > > > > BTW what makes it i440fx specific?
> > > > > Also, what happens if you try to use it
> > > > > with a different machine type?
> > > > Is is i440fx specific, please look at patch 22/28.
> > > > Also we have a specific check for i440fx, so CRS
> > > > will not be emitted for other machine types.
> > > >
> > > > Thanks,
> > > > Marcel
> > >
> > > In fact it won't work at all. Need to think about it,
> > > maybe we can make it work more generally.
> > > For CRS, should be possible to emit for q35 too?
> > We can make it work, but not on the scope of this series.
> > However, I'll add a IHostBridgeSnoop interface that will
> > make the device work only with associated bus and
> > this will make it less general.
> >
> > Thanks,
> > Marcel
>
> OK, this works too. Do you mean PCIHostBridgeSnoop?
Sure.

Thanks,
Marcel

> > > > > > +
> > > > > > +As opposed to PCI-2-PCI bridge's secondary bus, PXB's bus
> > > > > > +is a primary bus and can be associated with a NUMA node
> > > > > > +(different from the main host bridge) allowing the guest OS
> > > > > > +to recognize the proximity of a pass-through device to
> > > > > > +other resources as RAM and CPUs.
> > > > > > +
> > > > > > +Usage
> > > > > > +=====
> > > > > > +A detailed command line would be:
> > > > > > +
> > > > > > +[qemu-bin + storage options]
> > > > > > +-bios [seabios-dir]/out/bios.bin -L [seabios-dir]/out/
> > > > > > +-m 2G
> > > > > > +-object memory-backend-ram,size=1024M,policy=bind,host-nodes=0,id=ram-node0 
> > > > > > -numa node,nodeid=0,cpus=0,memdev=ram-node0
> > > > > > +-object 
> > > > > > memory-backend-ram,size=1024M,policy=interleave,host-nodes=0,id=ram-node1 -numa 
> > > > > > node,nodeid=1,cpus=1,memdev=ram-node1
> > > > > > +-device pxb-device,id=bridge1,bus=pci.0,numa_node=1,bus_nr=4 -netdev 
> > > > > > user,id=nd-device e1000,bus=bridge1,addr=0x4,netdev=nd
> > > > > > +-device pxb-device,id=bridge2,bus=pci.0,numa_node=0,bus_nr=8 -device 
> > > > > > e1000,bus=bridge2,addr=0x3
> > > > > > +-device pxb-device,id=bridge3,bus=pci.0,bus_nr=40 -drive 
> > > > > > if=none,id=drive0,file=[img] -device 
> > > > > > virtio-blk-pci,drive=drive0,scsi=off,bus=bridge3,addr=1
> > > > > > +
> > > > > > +Here you have:
> > > > > > + - 2 NUMA nodes for the guest, 0 and 1. (both mapped to the same NUMA node in 
> > > > > > host, but you can and should put it in different host NUMA nodes)
> > > > > > + - a pxb host bridge attached to NUMA 1 with an e1000 behind it
> > > > > > + - a pxb host bridge attached to NUMA 0 with an e1000 behind it
> > > > > > + - a pxb host bridge not attached to any NUMA with a hard drive behind it.
> > > > > > +
> > > > > > +Implementation
> > > > > > +==============
> > > > > > +The PXB is composed by:
> > > > > > +- HostBridge (TYPE_PXB_HOST)
> > > > > > +  The host bridge allows to register and query the PXB's rPCI root bus in QEMU.
> > > > > > +- PXBDev(TYPE_PXB_DEVICE)
> > > > > > +  It is a regular PCI Device that resides on the piix host-bridge bus and its 
> > > > > > bus uses the same PCI domain.
> > > > > > +  However, the bus behind is exposed through ACPI as a primary PCI bus and 
> > > > > > starts a new PCI hierarchy.
> > > > > > +  The interrupts from devices behind the PXB are routed through this device 
> > > > > > the same as if it were a
> > > > > > +  PCI-2-PCI bridge. The _PRT follows the i440fx model.
> > > > > > +- PCIBridgeDev(TYPE_PCI_BRIDGE_DEV)
> > > > > > +  Created automatically as part of init sequence.
> > > > > > +  When adding a device to PXB it is attached to the bridge for two reasons:
> > > > > > +  - Using the bridge will enable hotplug support
> > > > > > +  - All the devices behind the bridge will use bridge's IO/MEM windows 
> > > > > > compacting
> > > > > > +    the PCI address space.
> > > > > > +
> > > > > > --
> > > > > > 2.1.0