Re: [Qemu-devel] [Qemu-ppc] [PATCH v4 for-2.3 00/25] hw/pc: implement multiple primary busses for pc machines

[Marcel Apfelbaum]

On 03/10/2015 08:23 AM, Alexey Kardashevskiy wrote:
> On 03/10/2015 06:21 AM, Marcel Apfelbaum wrote:
> > On 03/09/2015 06:55 PM, Gerd Hoffmann wrote:
> > > On Mo, 2015-03-09 at 18:26 +0200, Marcel Apfelbaum wrote:
> > > > On 03/09/2015 04:19 PM, Gerd Hoffmann wrote:
> > > > >     Hi,
> > > > >
> > > > > > My series is based on commit 09d219a. Try please on top of this commit.
> > > > >
> > > > > Ok, that works.  Going to play with that now ;)
> > > > Good luck! ... and tell me what you think :)
> > > > If you need any help with the command line of the pxb device, let me know,.
> > >
> > > First thing I've noticed:  You need to define a numa node so you can
> > > pass a valid numa node to the pxb-device.  Guess that is ok as the whole
> > > point of this is to assign pci devices to numa nodes.  More complete
> > > test instructions would be nice though.
> > Exactly, this is by design. But you can also use it without specifying the
> > NUMA node...
> >
> > A detailed command line would be:
> >
> > [qemu-bin + storage options]
> > -bios [seabios-dir]/out/bios.bin -L [seabios-dir]/out/
> > -m 2G
> > -object memory-backend-ram,size=1024M,policy=bind,host-nodes=0,id=ram-node0
> > -numa node,nodeid=0,cpus=0,memdev=ram-node0
> > -object
> > memory-backend-ram,size=1024M,policy=interleave,host-nodes=0,id=ram-node1
> > -numa node,nodeid=1,cpus=1,memdev=ram-node1
> > -device pxb-device,id=bridge1,bus=pci.0,numa_node=1,bus_nr=4 -netdev
> > user,id=nd-device e1000,bus=bridge1,addr=0x4,netdev=nd
> > -device pxb-device,id=bridge2,bus=pci.0,numa_node=0,bus_nr=8 -device
> > e1000,bus=bridge2,addr=0x3
> > -device pxb-device,id=bridge3,bus=pci.0,bus_nr=40 -drive
> > if=none,id=drive0,file=[img] -device
> > virtio-blk-pci,drive=drive0,scsi=off,bus=bridge3,addr=1
>
>
> I replayed this patchset on top of 09d219a "acpi: update generated files" and 
> got this:
>
>
> qemu-system-x86_64: -object 
> memory-backend-ram,size=1024M,policy=bind,host-nodes=0,id=ram-node0: NUMA node 
> binding are not supported by this QEMU
> qemu-system-x86_64: -object 
> memory-backend-ram,size=1024M,policy=interleave,host-nodes=0,id=ram-node1: NUMA 
> node binding are not supported by this QEMU
Hi,

Please check your configuration (after you run ./configure script). See if you 
have a line like this:
- NUMA host support yes

> This is my exact command line:
>
> /scratch/alexey/p/qemu-build/x86_x86_64/x86_64-softmmu/qemu-system-x86_64 \
> -L /home/alexey/p/qemu/pc-bios/ \
> -hda x86/fc19_24GB_x86.qcow2 \
> -enable-kvm \
> -kernel x86/vmlinuz-3.12.11-201.fc19.x86_64 \
> -initrd x86/initramfs-3.12.11-201.fc19.x86_64.img \
> -append "root=/dev/sda3 console=ttyS0" \
> -nographic \
> -nodefaults \
> -chardev stdio,id=id2,signal=off,mux=on \
> -device isa-serial,id=id3,chardev=id2 \
> -mon id=id4,chardev=id2,mode=readline \
> -m 2G \
> -object memory-backend-ram,size=1024M,policy=bind,host-nodes=0,id=ram-node0 \
> -numa node,nodeid=0,cpus=0,memdev=ram-node0 \
> -object 
> memory-backend-ram,size=1024M,policy=interleave,host-nodes=0,id=ram-node1 \
> -numa node,nodeid=1,cpus=1,memdev=ram-node1 \
> -device pxb-device,id=bridge1,bus=pci.0,numa_node=1,bus_nr=4 \
> -netdev user,id=nd-device e1000,bus=bridge1,addr=0x4,netdev=nd \
> -device pxb-device,id=bridge2,bus=pci.0,numa_node=0,bus_nr=8 \
> -device e1000,bus=bridge2,addr=0x3 \
> -device pxb-device,id=bridge3,bus=pci.0,bus_nr=40 \
> -drive if=none,id=drive0,file=debian_lenny_powerpc_desktop.qcow2 \
> -device virtio-blk-pci,drive=drive0,scsi=off,bus=bridge3,addr=1 \
>
>
> What am I missing here?
See above, check for NUMA host support
> What I actually wanted to find out (instead of asking what I am doing now) is 
> is this PXB device a PCI device sitting on the same PCI host bus adapter (1) or 
> it is a separate PHB (2) with its own PCI
> domain (new XXXX in XXXX:00:00.0 PCI address)? I would think it is (1) but then what 
> exactly do you call "A primary PCI bus" here (that's my ignorance speaking, yes 
>  :) )? Thanks.
You are right, the PXB is a device on the piix host-bridge bus and its bus uses 
the same PCI domain.
However, the bus behind is exposed through ACPI as Primary PCI bus and starts a 
new PCI hierarchy.

You have a similar approach on Intel 450x chipset:
http://www.intel.com/design/chipsets/datashts/243771.htm
Look for 82454NX PCI Expander Bridge (PXB)

Thanks,
Marcel

> > Here you have:
> >   - 2 NUMA nodes for the guest, 0 and 1. (both mapped to the same NUMA node
> > in host, but you can and should put it in different host NUMA nodes)
> >   - a pxb host bridge attached to NUMA 1 with an e1000 behind it
> >   - a pxb host bridge attached to NUMA 0 with an e1000 behind it
> >   - a pxb host bridge not attached to any NUMA with a hard drive behind it.
> >
> > As you can see, since you already "decide" NUMA mapping at command line, it
> > is "natural" also to attach the pxbs to the NUMA nodes.
> >
> > > Second thing:  Booting with an unpatched seabios has bad effects:
> > >
> > > address@hidden ~]# cat /proc/iomem
> > > 00000000-000fffff : PCI Bus 0000:10
> > >    00000000-00000fff : reserved
> > >    00001000-0009fbff : System RAM
> > >    0009fc00-0009ffff : reserved
> > >    000c0000-000c91ff : Video ROM
> > >    000c9800-000ca1ff : Adapter ROM
> > >    000ca800-000ccbff : Adapter ROM
> > >    000f0000-000fffff : reserved
> > >      000f0000-000fffff : System ROM
> > > 00100000-3ffdffff : System RAM
> > >    01000000-0174bde4 : Kernel code
> > >    0174bde5-01d30cff : Kernel data
> > >    01eaa000-0202afff : Kernel bss
> > > 3ffe0000-3fffffff : reserved
> > > fd000000-fdffffff : 0000:00:02.0
> > >    fd000000-fdffffff : bochs-drm
> > > febc0000-febdffff : 0000:00:03.0
> > >    febc0000-febdffff : e1000
> > > febf0000-febf0fff : 0000:00:02.0
> > >    febf0000-febf0fff : bochs-drm
> > > fec00000-fec003ff : IOAPIC 0
> > > fed00000-fed003ff : HPET 0
> > >    fed00000-fed003ff : PNP0103:00
> > > fee00000-fee00fff : Local APIC
> > > feffc000-feffffff : reserved
> > > fffc0000-ffffffff : reserved
> > >
> > > "PCI Bus 0000:10" is bogus and "PCI Bus 0000:00" isn't there at all.
> > Yes, you shouldn't use pxb if you are not using the corresponding SeaBIOS.
> > However, as I understand we always attach a SeaBIOS binary with a QEMU
> > release,
> > so we should be OK with this.
> >
> > And this is the reason I wanted bios support *before* the PXB device
> > implementation,
> > but anyway, even if we have them in the same time, as long as the release
> > has both pxb and BIOS with pxb support, is OK. (I think...)
> >
> > I appreciate you looking into this and if you need further assistance
> > don't hesitate to mail me! :)
> >
> > Thanks,
> > Marcel
> >
> > > cheers,
> > >    Gerd