Re: [Qemu-devel] snabbswitch integration with QEMU for userspace ethernet I/O

[Michael S. Tsirkin]

On Mon, May 27, 2013 at 12:01:07PM -0500, Anthony Liguori wrote:
> Paolo Bonzini <address@hidden> writes:
> 
> > Il 27/05/2013 18:18, Anthony Liguori ha scritto:
> >> Paolo Bonzini <address@hidden> writes:
> >> 
> >>> Il 27/05/2013 11:34, Stefan Hajnoczi ha scritto:
> >>>> On Sun, May 26, 2013 at 11:32:49AM +0200, Luke Gorrie wrote:
> >>>>> Stefan put us onto the highly promising track of vhost/virtio. We have
> >>>>> implemented this between Snabb Switch and the Linux kernel, but not
> >>>>> directly between Snabb Switch and QEMU guests. The "roadblock" we have 
> >>>>> hit
> >>>>> is embarrasingly basic: QEMU is using user-to-kernel system calls to 
> >>>>> setup
> >>>>> vhost (open /dev/net/tun and /dev/vhost-net, ioctl()s) and I haven't 
> >>>>> found
> >>>>> a good way to map these towards Snabb Switch instead of the kernel.
> >>>>
> >>>> vhost_net is about connecting the a virtio-net speaking process to a
> >>>> tun-like device.  The problem you are trying to solve is connecting a
> >>>> virtio-net speaking process to Snabb Switch.
> >>>>
> >>>> Either you need to replace vhost or you need a tun-like device
> >>>> interface.
> >>>>
> >>>> How does your switch talk to hardware?
> >>>
> >>> And also, is your switch monolithic or does it consist of different
> >>> processes?
> >>>
> >>> If you already have processes talking to each other, the first thing
> >>> that came to my mind was a new network backend, similar to net/vde.c but
> >>> more featureful (so that you support the virtio headers for offloading,
> >>> for example).  Then you would use "-netdev snabb,id=net0 -device
> >>> e1000,netdev=net0".
> >> 
> >> It would be very interesting to combine this with vmsplice/splice.
> >
> > Was zero-copy vmsplice/splice actually ever implemented?  I thought it
> > was reverted.
> 
> Not sure what context you're talking about re: zero copy...  a pipe can
> store references to pages instead of having a buffer that stores data.
> That certainly is there today--otherwise the interface is pointless.
> 
> When splicing from pipe to pipe, you can move those references without
> copying the data.
> 
> When vmsplicing from a userspace region to a pipe, the kernel just
> stores references to the pages.  vmsplicing from a pipe to userspace
> OTOH will copy the data.  This is fixable at least when dealing with
> GIFT'd pages.  For guest-to-guest traffic, you wouldn't be gifting the
> pages I don't think.
> 
> For implementing guest-to-guest traffic, the source QEMU can vmsplice
> the packet to a pipe that is shared with the vswitch.  The vswitch can
> tee(3) the first N bytes to a second pipe such that it can read the
> info needed for routing decisions.
> 
> Once the decision is made, if it's a local guest, it can splice() the
> packet to the appropriate destination QEMU process or another vswitch
> daemon (no data copy here).
> 
> Finally, the destination QEMU process can vmsplice() from the pipe which
> will copy the data (this is the only copy).

AFAIK splice is mostly useless for networking as there's no way to
get notified when packet has been sent.

> If vswitch needs to route externally, then it would need to splice() to
> a macvtap.
> 
> macvtap should be able to send the packet without copying the data.  Not
> sure that this last work will work as expected but if it doesn't, that's
> a bug that can/should be fixed.
> 
> The kernel cannot do better than the above modulo any overhead from
> userspace context switching[*].

Also modulo scheduler latency - kernel processes packets
in interrupt context. There's a reason e.g. OVS runs data-path in
kernel.

>  Guest-to-guest requires a copy.
> Normally macvtap is undesirable because it's tightly connected to a
> network adapter but that is a desirable trait in this case.
> 
> N.B., I'm not advocating making all switching decisions in
> userspace. Just pointing out how it can be done efficiently.
> 
> [*] in theory the kernel could do zero copy receive but i'm not sure
> it's feasible in practice.
> 
> Regards,
> 
> Anthony Liguori
> 
> >
> > Paolo
> >
> >>> It would be slower than vhost-net, for example no zero-copy
> >>> transmission.
> >> 
> >> With splice, I think you could at least get single copy guest-to-guest
> >> networking which is about as good as can be done.
> >> 
> >> Regards,
> >> 
> >> Anthony Liguori
> >> 
> >>>> 3. Use the kernel as a middle-man. Create a double-ended "veth"
> >>>> interface and have Snabb Switch and QEMU each open a PF_PACKET
> >>>> socket and accelerate it with VHOST_NET.
> >>>
> >>> As Michael, mentioned, this could be macvtap on the interface that you
> >>> have already created in the switch and passed to vhost-net.  Then you do
> >>> not have to do anything in QEMU.
> >>>
> >>> Paolo
> >>>
> >>>> If you are using the Linux network stack then it might be better to
> >>>> integrate with vhost maybe as a tun-like device driver.
> >>>>
> >>>> Stefan
> >>>>
> >>>>