Re: [Qemu-devel] [RFC v5 0/6] pci_expander_brdige: support separate pci domain for pxb-pcie

[Zihan Yang]

Hi Marcel,

Marcel Apfelbaum <address@hidden> 于2018年9月20日周四 下午2:39写道：
>
>   Hi Zihan,
>
> On 09/18/2018 04:41 PM, Michael S. Tsirkin wrote:
> > Cc Laine, Eric for an opinion about the management interface.
> >
> > On Mon, Sep 17, 2018 at 10:57:31PM +0800, Zihan Yang wrote:
> >> Hi all
> >>
> >> Here is a minimal working version of supporting multiple pci domains.
> >> The next a few paragraphs will illustrate the purpose and use example.
> >> Current issue and limitations will be in last 2 paragraphs, followed
> >> by the changelog of each verison.
> >>
> >> Currently only q35 host bridge is allocated an item in MCFG table, all
> >> pxb-pcie host bridges stay within pci domain 0. This series of patches
> >> allow each pxb-pcie to be put in separate pci domain, allocating a new
> >> MCFG table item for it.
> >>
> >> Users can configure whether to put pxb host bridge into separate domain
> >> by specifying property 'domain_nr' of pxb-pcie device. 'bus_nr' property
> >> indicates the Base Bus Number(BBN) of the pxb-pcie host bridge. Another
> >> property max_bus specifies the maximum desired bus numbers to reduce
> >> MCFG space cost. Example command is
> >>
> >>      -device pxb-pcie,id=bridge3,bus="pcie.0",domain_nr=1,max_bus=15
> >>
> >> Then this pxb-pcie host bridge is placed at pci domain 1, and only reserve
> >> (15+1)=16 buses, which is much smaller than the default 256 buses.
> >>
> >> Compared with previous version, this version is much simpler because
> >> mcfg of extra domain now has a relatively fixed address, as suggested
> >> by Marcel and Gerd. Putting extra mmconfig above 4G and let seabios
> >> leave them for guest os will be expected in next version. The range is
> >> [0x80000000, 0xb0000000), which allows us to hold 4x busses compared
> >> with before.
> >>
> >> A complete command line for test is follows, you need to replace 
> >> GUEST_IMAGE,
> >> DATA_IMAGE and SEABIOS_BIN with proper environment variable
> >>
> >> ./x86_64-softmmu/qemu-system-x86_64 \
> >>      -machine q35,accel=kvm -smp 2 -m 2048 \
> >>      -drive file=${GUEST_IMAGE}  -netdev user,id=realnet0 \
> >>      -device e1000e,netdev=realnet0,mac=52:54:00:12:34:56 \
> >>      -device pxb-pcie,id=bridge3,bus="pcie.0",domain_nr=1 \
> >>      -device pcie-root-port,id=rp1,bus=bridge3,addr=1c.0,port=8,chassis=8 \
> >>      -drive if=none,id=drive0,file=${DATA_IMAGE} \
> >>      -device virtio-scsi-pci,id=scsi,bus=rp1,addr=00.0 \
> >>      -bios ${SEABIOS_BIN}
> >>
> >> There are a few limitations, though
> >> 1. Legacy interrupt routing is not dealt with yet. There is only support 
> >> for
> >>     devices using MSI/MSIX
> > That's probably a must have. What makes it hard to add?
>
> Zihan, can you please elaborate on what is the exact problem?
> pxb-pcie devices placed in PCI domain 0 do support
> legacy  interrupts, the question is what is different for multiple
> PCI domains.

Sorry for the delay. One problem I know of interrupt is that pxb-pcie in domain
0 directly maps irq to pci root of q35 host. In multiple domains they are under
different pci root, so we need to change the routing rule and modify PRT
table of pxb pcie host to route interrupt to ich9. But pxb does not have ISA
under it, I'm not sure whether this would be a problem.

It seems to require some engineering effort, but first I'm trying to figure out
another issue: why the same device under domain 1 will also appear in domain 0.
I find there are many resource allocation failure for device under domain 1,
dmesg shows

[    0.179664] pci 0001:00:00.0: BAR 14: no space for [mem size 0x00100000]
[    0.179667] pci 0001:00:00.0: BAR 14: failed to assign [mem size 0x00100000]
[    0.179668] pci 0001:00:00.0: BAR 13: no space for [io  size 0x1000]
[    0.179670] pci 0001:00:00.0: BAR 13: failed to assign [io  size 0x1000]
[    0.179672] pci 0001:00:00.0: BAR 0: no space for [mem size 0x00000100 64bit]
[    0.179673] pci 0001:00:00.0: BAR 0: failed to assign [mem size
0x00000100 64bit]
[    0.179676] pci 0001:01:10.0: BAR 6: no space for [mem size 0x00040000 pref]
[    0.179678] pci 0001:01:10.0: BAR 6: failed to assign [mem size
0x00040000 pref]
[    0.179680] pci 0001:01:10.0: BAR 0: no space for [mem size 0x00020000]
[    0.179681] pci 0001:01:10.0: BAR 0: failed to assign [mem size 0x00020000]

A humble guess is that the pxb host bus is a child bus of pxb-pcie, which might
be scanned twice during initialization of q35 and pxb host bus and causes the
potential resource conflict. But that is just a guess now, not quite
sure it is the
issue of qemu or seabios. I will try to find out.

> Thanks,
> Marcel
>
> >
>
> [...]