[Qemu-devel] Re: [PATCH v5 0/4] virtio: Use ioeventfd for virtqueue notify

[Stefan Hajnoczi]

On Mon, Dec 13, 2010 at 6:52 PM, Michael S. Tsirkin <address@hidden> wrote:
> On Mon, Dec 13, 2010 at 05:57:28PM +0000, Stefan Hajnoczi wrote:
>> On Mon, Dec 13, 2010 at 4:28 PM, Stefan Hajnoczi <address@hidden> wrote:
>> > On Mon, Dec 13, 2010 at 4:12 PM, Michael S. Tsirkin <address@hidden> wrote:
>> >> On Mon, Dec 13, 2010 at 03:27:06PM +0000, Stefan Hajnoczi wrote:
>> >>> On Mon, Dec 13, 2010 at 1:36 PM, Michael S. Tsirkin <address@hidden> 
>> >>> wrote:
>> >>> > On Mon, Dec 13, 2010 at 03:35:38PM +0200, Michael S. Tsirkin wrote:
>> >>> >> On Mon, Dec 13, 2010 at 01:11:27PM +0000, Stefan Hajnoczi wrote:
>> >>> >> > Fresh results:
>> >>> >> >
>> >>> >> > 192.168.0.1 - host (runs netperf)
>> >>> >> > 192.168.0.2 - guest (runs netserver)
>> >>> >> >
>> >>> >> > host$ src/netperf -H 192.168.0.2 -- -m 200
>> >>> >> >
>> >>> >> > ioeventfd=on
>> >>> >> > TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.0.2
>> >>> >> > (192.168.0.2) port 0 AF_INET
>> >>> >> > Recv   Send    Send
>> >>> >> > Socket Socket  Message  Elapsed
>> >>> >> > Size   Size    Size     Time     Throughput
>> >>> >> > bytes  bytes   bytes    secs.    10^6bits/sec
>> >>> >> >  87380  16384    200    10.00    1759.25
>> >>> >> >
>> >>> >> > ioeventfd=off
>> >>> >> > TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.0.2
>> >>> >> > (192.168.0.2) port 0 AF_INET
>> >>> >> > Recv   Send    Send
>> >>> >> > Socket Socket  Message  Elapsed
>> >>> >> > Size   Size    Size     Time     Throughput
>> >>> >> > bytes  bytes   bytes    secs.    10^6bits/sec
>> >>> >> >
>> >>> >> >  87380  16384    200    10.00    1757.15
>> >>> >> >
>> >>> >> > The results vary approx +/- 3% between runs.
>> >>> >> >
>> >>> >> > Invocation:
>> >>> >> > $ x86_64-softmmu/qemu-system-x86_64 -m 4096 -enable-kvm -netdev
>> >>> >> > type=tap,id=net0,ifname=tap0,script=no,downscript=no -device
>> >>> >> > virtio-net-pci,netdev=net0,ioeventfd=on|off -vnc :0 -drive
>> >>> >> > if=virtio,cache=none,file=$HOME/rhel6-autobench-raw.img
>> >>> >> >
>> >>> >> > I am running qemu.git with v5 patches, based off
>> >>> >> > 36888c6335422f07bbc50bf3443a39f24b90c7c6.
>> >>> >> >
>> >>> >> > Host:
>> >>> >> > 1 Quad-Core AMD Opteron(tm) Processor 2350 @ 2 GHz
>> >>> >> > 8 GB RAM
>> >>> >> > RHEL 6 host
>> >>> >> >
>> >>> >> > Next I will try the patches on latest qemu-kvm.git
>> >>> >> >
>> >>> >> > Stefan
>> >>> >>
>> >>> >> One interesting thing is that I put virtio-net earlier on
>> >>> >> command line.
>> >>> >
>> >>> > Sorry I mean I put it after disk, you put it before.
>> >>>
>> >>> I can't find a measurable difference when swapping -drive and -netdev.
>> >>
>> >> One other concern I have is that we are apparently using
>> >> ioeventfd for all VQs. E.g. for virtio-net we probably should not
>> >> use it for the control VQ - it's a waste of resources.
>> >
>> > One option is a per-device (block, net, etc) bitmap that masks out
>> > virtqueues.  Is that something you'd like to see?
>> >
>> > I'm tempted to mask out the RX vq too and see how that affects the
>> > qemu-kvm.git specific issue.
>>
>> As expected, the rx virtqueue is involved in the degradation.  I
>> enabled ioeventfd only for the TX virtqueue and got the same good
>> results as userspace virtio-net.
>>
>> When I enable only the rx virtqueue, performs decreases as we've seen above.
>>
>> Stefan
>
> Interesting. In particular this implies something's wrong with the
> queue: we should not normally be getting notifications from rx queue
> at all. Is it running low on buffers? Does it help to increase the vq
> size?  Any other explanation?

I made a mistake, it is the *tx* vq that causes reduced performance on
short packets with ioeventfd.  I double-checked the results and the rx
vq doesn't affect performance.

Initially I thought the fix would be to adjust the tx mitigation
mechanism since ioeventfd does its own mitigation of sorts.  Multiple
eventfd signals will be coalesced into one qemu-kvm event handler call
if qemu-kvm didn't have a chance to handle the first event before the
eventfd was signalled again.

I added -device virtio-net-pci tx=immediate to flush the TX queue
immediately instead of scheduling a BH or timer.  Unfortunately this
had little measurable effect and performance stayed the same.  This
suggests most of the latency is between the guest's pio write and
qemu-kvm getting around to handling the event.

You mentioned that vhost-net has the same performance issue on this
benchmark.  I guess a solution for vhost-net may help virtio-ioeventfd
and vice versa.

Are you happy with this patchset if I remove virtio-net-pci
ioeventfd=on|off so only virtio-blk-pci has ioeventfd=on|off (with
default on)?  For block we've found it to be a win and the initial
results looked good for net too.

Stefan