Re: [RFC PATCH 01/12] configure: Add iovisor/ubpf project as a submodule for QEMU

[Thomas Huth]

On 20/06/2022 11.29, Zhang, Chen wrote:
> > -----Original Message-----
> > From: Thomas Huth <thuth@redhat.com>
> > Sent: Monday, June 20, 2022 4:47 PM
> > To: Daniel P. Berrangé <berrange@redhat.com>; Zhang, Chen
> > <chen.zhang@intel.com>
> > Cc: Jason Wang <jasowang@redhat.com>; qemu-dev <qemu-
> > devel@nongnu.org>; Paolo Bonzini <pbonzini@redhat.com>; Eduardo
> > Habkost <eduardo@habkost.net>; Eric Blake <eblake@redhat.com>; Markus
> > Armbruster <armbru@redhat.com>; Peter Maydell
> > <peter.maydell@linaro.org>; Laurent Vivier <lvivier@redhat.com>; Yuri
> > Benditovich <yuri.benditovich@daynix.com>; Andrew Melnychenko
> > <andrew@daynix.com>
> > Subject: Re: [RFC PATCH 01/12] configure: Add iovisor/ubpf project as a
> > submodule for QEMU
> >
> > On 20/06/2022 10.11, Daniel P. Berrangé wrote:
> > > On Mon, Jun 20, 2022 at 05:59:06AM +0000, Zhang, Chen wrote:
> > > > > -----Original Message-----
> > > > > From: Daniel P. Berrangé <berrange@redhat.com>
> > > > > Sent: Friday, June 17, 2022 4:05 PM
> > > > > To: Zhang, Chen <chen.zhang@intel.com>
> > > > > Cc: Jason Wang <jasowang@redhat.com>; qemu-dev <qemu-
> > > > > devel@nongnu.org>; Paolo Bonzini <pbonzini@redhat.com>; Eduardo
> > > > > Habkost <eduardo@habkost.net>; Eric Blake <eblake@redhat.com>;
> > > > > Markus Armbruster <armbru@redhat.com>; Peter Maydell
> > > > > <peter.maydell@linaro.org>; Thomas Huth <thuth@redhat.com>;
> > Laurent
> > > > > Vivier <lvivier@redhat.com>; Yuri Benditovich
> > > > > <yuri.benditovich@daynix.com>; Andrew Melnychenko
> > > > > <andrew@daynix.com>
> > > > > Subject: Re: [RFC PATCH 01/12] configure: Add iovisor/ubpf project
> > > > > as a submodule for QEMU
> > > > >
> > > > > On Fri, Jun 17, 2022 at 03:36:19PM +0800, Zhang Chen wrote:
> > > > > > Make iovisor/ubpf project be a git submodule for QEMU.
> > > > > > It will auto clone ubpf project when configure QEMU.
> > > > >
> > > > > I don't think we need todo this. As it is brand new functionality we
> > > > > don't have any back compat issues. We should just expect the distros
> > > > > to ship ubpf if they want their QEMU builds to take advantage of it.
> > > >
> > > > Yes, agree. It's the best way to use the uBPF project.
> > > > But current status is distros(ubuntu, RHEL...) does not ship the
> > > > iovisor/ubpf like the iovisor/bcc. So I have to do it.
> > > > Or do you have any better suggestions?
> > >
> > > If distros want to support the functionality, they can add packages
> > > for it IMHO.
> >
> > Yes, let's please avoid new submodules. Submodules can sometimes be a
> > real PITA (e.g. if you forget to update before rsync'ing your code to a
> > machine that has limited internet access), and if users install QEMU from
> > sources, they can also install ubpf from sources, too.
> > And if distros want to support this feature, they can package ubpf on their
> > own, as Daniel said.
>
> Hi Daniel and Thomas,
>
> I don't know much the background history of QEMU submodules, but meson build
> is a submodule for QEMU too. It means user can't install QEMU from sources
> with limited internet access.

There is no written policy, but I think the general consensus is that we 
only ship code in submodules if:

1) It's not available in a required version in distros yet

and

2) it is essentially required to build QEMU (like meson) or if the feature 
has been part of the QEMU sources before and then moved to a separate 
repository (like slirp).

We ship meson as a submodule since we require some meson features that are 
not available with the meson versions in the distros yet. Once the distros 
catch up, we'll likely remove the meson submodule from QEMU.

> And back to Daniel's comments,  Yes, the best way is distros add the ubpf 
> packages,
> But maybe it's too late to implement new features for us. We can introduce the 
> submodule now and
> auto change to the distros's lib when distros add it.  For example QEMU's 
> submodule SLIRP do it in the same way.

slirp used to be part of the QEMU repository, but then has been moved to a 
separate project a while ago. However, at that point in time there weren't 
any packages ins distros yet, so we had to include it as a submodule.

Now that the distros ship it, too, we're planning to remove the slirp 
submodule from QEMU soon, see:

 https://lists.gnu.org/archive/html/qemu-devel/2022-04/msg00974.html

> It make user experience the latest technology
> with no other dependencies.

Well, that's only true if we update the submodule in QEMU regularly. If we 
forget to update, we could easily miss some important (maybe even security 
related) fixes from the upstream projects. This can be a nightmare for 
distros, when they then have to go around and look into each and every 
projects whether they embed a certain code module that needs a CVE fix. It's 
better if it can be fixed in one central spot instead.

> uBPF infrastructure have the ability to extend the capabilities without 
> requiring
> changing source code. If we not allow it, we have to re-implement all the eBPF 
> assembler, disassembler,
> interpreter, and JIT compiler like DPDK userspace eBPF support (DPDK can't use 
> ubpf project by license issue).

Not sure whether I understood that statement right ... nobody said that QEMU 
should not allow it - we just suggested to rely on a system installation of 
ubpf instead of embedding the code. Or is that not possible?? (I don't know 
that project yet - isn't it possible to compile it as a shared library?)

 Thomas