Re: [PATCH RFC v2 00/12] virtio-net: add support for SR-IOV emulation

[Akihiko Odaki]

On 2023/12/11 16:26, Jason Wang wrote:
> On Mon, Dec 11, 2023 at 1:30 PM Akihiko Odaki <akihiko.odaki@daynix.com> wrote:
> > On 2023/12/11 11:52, Jason Wang wrote:
> > > On Sun, Dec 10, 2023 at 12:06 PM Akihiko Odaki <akihiko.odaki@daynix.com> wrote:
> > > > Introduction
> > > > ------------
> > > >
> > > > This series is based on the RFC series submitted by Yui Washizu[1].
> > > > See also [2] for the context.
> > > >
> > > > This series enables SR-IOV emulation for virtio-net. It is useful
> > > > to test SR-IOV support on the guest, or to expose several vDPA devices
> > > > in a VM. vDPA devices can also provide L2 switching feature for
> > > > offloading though it is out of scope to allow the guest to configure
> > > > such a feature.
> > > >
> > > > The PF side code resides in virtio-pci. The VF side code resides in
> > > > the PCI common infrastructure, but it is restricted to work only for
> > > > virtio-net-pci because of lack of validation.
> > > >
> > > > User Interface
> > > > --------------
> > > >
> > > > A user can configure a SR-IOV capable virtio-net device by adding
> > > > virtio-net-pci functions to a bus. Below is a command line example:
> > > >     -netdev user,id=n -netdev user,id=o
> > > >     -netdev user,id=p -netdev user,id=q
> > > >     -device pcie-root-port,id=b
> > > >     -device virtio-net-pci,bus=b,addr=0x0.0x3,netdev=q,sriov-pf=f
> > > >     -device virtio-net-pci,bus=b,addr=0x0.0x2,netdev=p,sriov-pf=f
> > > >     -device virtio-net-pci,bus=b,addr=0x0.0x1,netdev=o,sriov-pf=f
> > > >     -device virtio-net-pci,bus=b,addr=0x0.0x0,netdev=n,id=f
> > > >
> > > > The VFs specify the paired PF with "sriov-pf" property. The PF must be
> > > > added after all VFs. It is user's responsibility to ensure that VFs have
> > > > function numbers larger than one of the PF, and the function numbers
> > > > have a consistent stride.
> > >
> > > This seems not user friendly. Any reason we can't just allow user to
> > > specify the stride here?
> >
> > It should be possible to assign addr automatically without requiring
> > user to specify the stride. I'll try that in the next version.
> >
> > > Btw, I vaguely remember qemu allows the params to be accepted as a
> > > list. If this is true, we can accept a list of netdev here?
> >
> > Yes, rocker does that. But the problem is not just about getting
> > parameters needed for VFs, which I forgot to mention in the cover letter
> > and will explain below.
> >
> > > > Keeping VF instances
> > > > --------------------
> > > >
> > > > A problem with SR-IOV emulation is that it needs to hotplug the VFs as
> > > > the guest requests. Previously, this behavior was implemented by
> > > > realizing and unrealizing VFs at runtime. However, this strategy does
> > > > not work well for the proposed virtio-net emulation; in this proposal,
> > > > device options passed in the command line must be maintained as VFs
> > > > are hotplugged, but they are consumed when the machine starts and not
> > > > available after that, which makes realizing VFs at runtime impossible.
> > >
> > > Could we store the device options in the PF?
> >
> > I wrote it's to store the device options, but the problem is actually
> > more about realizing VFs at runtime instead of at the initialization time.
> >
> > Realizing VFs at runtime have two major problems. One is that it delays
> > the validations of options; invalid options will be noticed when the
> > guest requests to realize VFs.
>
> If PCI spec allows the failure when creating VF, then it should not be
> a problem.

I doubt the spec cares such a failure at all. VF enablement should 
always work for a real hardware. It's neither user-friendly to tell 
configuration errors at runtime.

> > netdevs also warn that they are not used
> > at initialization time, not knowing that they will be used by VFs later.
>
> We could invent things to calm down this false positive.
>
> > References to other QEMU objects in the option may also die before VFs
> > are realized.
>
> Is there any other thing than netdev we need to consider?

You will also want to set a distinct mac for each VF. Other properties 
does not matter much in my opinion.

> > The other problem is that QEMU cannot interact with the unrealized VFs.
> > For example, if you type "device_add virtio-net-pci,id=vf,sriov-pf=pf"
> > in HMP, you will expect "device_del vf" works, but it's hard to
> > implement such behaviors with unrealized VFs.
>
> I think hotplug can only be done at PF level if we do that.

Assuming you mean to let netdev and mac accept arrays, yes.

> > I was first going to compromise and allow such quirky behaviors, but I
> > realized such a compromise is unnecessary if we reuse the PCI power down
> > logic so I wrote v2.
>
> Haven't checked the code, but anything related to the PM here?

You mean power management? We don't have to care about PCI power down 
for VFs because powering down a SR-IOV PCI device will reset it and thus 
disable its VFs.

Regards,
Akihiko Odaki