Re: [Qemu-devel] [QA-virtio]:Why vring size is limited to 1024?

[Michael S. Tsirkin]

On Wed, Oct 08, 2014 at 03:17:56PM +0800, Zhangjie (HZ) wrote:
> Thanks for your patient answer! :-)
> 
> On 2014/9/30 17:33, Michael S. Tsirkin wrote:
> > On Tue, Sep 30, 2014 at 04:36:00PM +0800, Zhangjie (HZ) wrote:
> >> Hi,
> >> There exits packets loss when we do packet forwarding in VM,
> >> especially when we use dpdk to do the forwarding. By enlarging vring
> >> can alleviate the problem.
> > 
> > I think this has to do with the fact that dpdk disables
> > checksum offloading, this has the side effect of disabling
> > segmentation offloading.
> > 
> > Please fix dpdk to support checksum offloading, and
> > I think the problem will go away.
> In some application scene, loss of udp packets are not allowed,
>  and udp packets are always short than mtu.
> So, we need to support high pps(eg.0.3M Packets/s) forwarding, and
> offloading cannot fix it.

That's the point. With UFO you get larger than MTU UDP packets:
http://www.linuxfoundation.org/collaborate/workgroups/networking/ufo

Additionally, checksum offloading reduces CPU utilization
and reduces the number of data copies, allowing higher pps
with smaller buffers.

It might look like queue depth helps performance for netperf, but in
real-life workloads the latency under load will suffer, with more
protocols implementing tunnelling on top of UDP such extreme bufferbloat
will not be tolerated.

> > 
> > 
> >> But now vring size is limited to 1024 as follows:
> >> VirtQueue *virtio_add_queue(VirtIODevice *vdev, int queue_size,
> >>                             void (*handle_output)(VirtIODevice *, 
> >> VirtQueue *))
> >> {
> >>    ...
> >>    if (i == VIRTIO_PCI_QUEUE_MAX || queue_size > VIRTQUEUE_MAX_SIZE)
> >>         abort();
> >> }
> >> ps:#define VIRTQUEUE_MAX_SIZE 1024
> >> I delete the judgement code, and set vring size to 2048,
> >> VM can be successfully started, and the network is ok too.
> >> So, Why vring size is limited to 1024 and what is the influence?
> >>
> >> Thanks!
> > 
> > There are several reason for this limit.
> > First guest has to allocate descriptor buffer which is 16 * vring size.
> > With 1K size that is already 16K which might be tricky to
> > allocate contigiously if memory is fragmented when device is
> > added by hotplug.
> That is very
> > The second issue is that we want to be able to implement
> > the device on top of linux kernel, and
> > a single descriptor might use all of
> > the virtqueue. In this case we wont to be able to pass the
> > descriptor directly to linux as a single iov, since
> > that is limited to 1K entries.
> For the second issue, I wonder if it is ok to set vring size of virtio-net to 
> large than 1024,
> as for net work, there is at most 18 pages for a skb, it will not exceed iov.
> > 
> >> -- 
> >> Best Wishes!
> >> Zhang Jie
> > .
> > 
> 
> -- 
> Best Wishes!
> Zhang Jie