Re: [Qemu-devel] >256 Virtio-net-pci hotplug Devices

[Marcel Apfelbaum]

On 18/07/2017 0:50, Kinsella, Ray wrote:
> Hi folks,

Hi Ray,

> I am trying to create a VM that supports hot-plugging a large number of 
> virtio-net-pci device,
> up to 1000 devices initially.
>
>  From the docs (see below) and from playing with QEMU, it looks like there are 
> two options.
> Both with limitations.
>
> PCI Express switch
>
> It looks like using a PCI Express switch hierarchy is not an option due to bus 
> exhaustion.
> Each downstream port creates a separate bus and each downstream port only 
> supports hot-plugging a single device.
> So this gives us a max of 256-ish buses/devices pairs.

Right.

> PCI Root Ports
>
> The other option is use a flatter hierarchy, with a number of multi-function 
> PCI Root Ports hanging off 'pcie.0'.
> However each 'PCI Root Port' can support hot-plugging a single device.
> So this method really becomes a function of how many free address we have on 
> 'pci.0'.
> If we make room for say 16 multifunction devices, we get 16*8 ... 128
> So ultimately, this will approach will give us a similar to number to using a 
> switch.
>
> Is there another method?

You can use multi-function PCIe Root Ports, this will give you 8 ports
per slot, if you have 16 empty slots (I think we have more) you reach
128 root ports.
Then you can use multi-function  virtio-net-pci devices, this will
give you 8 functions per port, so you reach the target of 1024 devices.

You loose hot-plug granularity since you can hot-plug 8-functions group,
but maybe is OK, depending on your scenario.

> ( pxb-pcie doesn't support hotplug for instance, and only a single pcie domain 
> is supported qemu )

Even so, you can use one cold-plugged pxb-pcie if you don't
have enough empty slots on pcie.0, in order to reach the maximum
number of PCIe Root Ports (256) which is the maximum for a single
PCI domain.

If you need granularity per single device (1000+ hot-pluggable),
you could enhance the pxb-pcie to support multiple pci domains.

Thanks,
Marcel


> Thanks,
>
> Ray K
>
> Pcie.txt,
> ( https://github.com/qemu/qemu/blob/master/docs/pcie.txt )
> Q35 preso
> ( http://wiki.qemu.org/images/4/4e/Q35.pdf )
>
>