Re: [Qemu-devel] [PATCH] Tap: fix vcpu long time io blocking on tap

[Wangkai (Kevin,C)]

> -----Original Message-----
> From: Stefan Hajnoczi [mailto:address@hidden
> Sent: Monday, July 14, 2014 4:43 PM
> To: Wangkai (Kevin,C)
> Cc: address@hidden; address@hidden; Lee yang
> Subject: Re: [PATCH] Tap: fix vcpu long time io blocking on tap
> 
> On Mon, Jul 14, 2014 at 01:55:05AM +0000, Wangkai (Kevin,C) wrote:
> >
> >
> > > -----Original Message-----
> > > From: Stefan Hajnoczi [mailto:address@hidden
> > > Sent: Friday, July 11, 2014 9:04 PM
> > > To: Wangkai (Kevin,C)
> > > Cc: address@hidden; address@hidden; Lee yang
> > > Subject: Re: [PATCH] Tap: fix vcpu long time io blocking on tap
> > >
> > > On Fri, Jul 11, 2014 at 01:05:30AM +0000, Wangkai (Kevin,C) wrote:
> > > > When used a tap as net driver for vm, if too many packets was
> > > > delivered to the guest os via tap interface, the guest os will be
> > > > blocked on io events for a long time, while tap driver was
> busying
> > > process packets.
> > > >
> > > > kvm vcpu thread block on io lock call trace:
> > > >   __lll_lock_wait
> > > >   _L_lock_1004
> > > >   __pthread_mutex_lock
> > > >   qemu_mutex_lock
> > > >   kvm_cpu_exec
> > > >   qemu_kvm_cpu_thread_fn
> > > >   start_thread
> > > >
> > > > qemu io thread call trace:
> > > >   ...
> > > >   qemu_net_queue_send
> > > >   tap_send
> > > >   qemu_iohandler_poll
> > > >   main_loop_wait
> > > >   main_loop
> > > >
> > > >
> > > > I think the qemu io lock time should be as small as possible, and
> > > > the io work slice should be limited at a particular ration or
> time.
> > > >
> > > > ---
> > > > Signed-off-by: Wangkai <address@hidden>
> > >
> > > How many packets are you seeing in a single tap_send() call?
> > >
> > > Have you profiled the tap_send() code path?  Maybe it is performing
> > > some operation that is very slow.
> > >
> > > By the way, if you want good performance you should use vhost_net
> > > instead of userspace vhost_net.  Userspace virtio-net is not very
> > > optimized.
> > >
> > > Stefan
> >
> >
> > Hi Stefan,
> >
> > I am not use profile, just debug with gdb and code review.
> 
> It's worth understanding the root cause for this behavior because
> something is probably wrong.
> 
> > When packets delivered, I found the VM was hung, and I check qemu run
> >
> > State by gdb, I see the call trace for IO thread and vcpu thread, and
> >
> > I add debug info to check how many packets within tap_send, the info
> below:
> >
> > total recv 393520 time 1539821 us
> > total recv 1270 time 4931 us
> > total recv 257872 time 995828 us
> > total recv 10745 time 41438 us
> > total recv 505387 time 2000925 us
> 
> 505387 packets or 505387 bytes?
> 
> If that's packets, then even with small 64-byte packets that would mean
> 32 MB of pending data!
> 
> Are you running a networking benchmark where you'd expect lots of
> packets?
> 
> Have you checked how lot the time between tap_send() calls is?  Perhaps
> something else in QEMU is blocking the event loop so packets accumulate
> in the host kernel.
> 
> Stefan

Hi Stefan,

Here the detail network:

+--------------------------------------------+
| The host add tap1 and eth10 to bridge 'br1'|                     +--------+
| +------------+                             |                     |  send  |
| |   VM  eth1-+-tap1 --- bridge --- eth10 --+---------------------+ packets|
| +------------+                             |                     |        |
+--------------------------------------------+                     +--------+

Qemu start vm by virtio, use tap interface, option is:
-net nic,vlan=101, model=virtio -net 
tap,vlan=101,ifname=tap1,script=no,downscript=no

And add tap1 and eth10 to bridge br1 in the host:
Brctl addif br1 tap1
Brctl addif br1 eth10

total recv 505387 time 2000925 us:
means call tap_send once dealing 505387 packets, the packet payload was 300 
bytes, and
time use for tap_send() was 2,000,925 micro-seconds, time was measured by 
record time stamp
at function tap_send() start and end.

We just test the performance of VM.

Regards
Wangkai