Re: [Qemu-devel] [PATCH] Tap: fix vcpu long time io blocking on tap

[Wangkai (Kevin,C)]

> -----Original Message-----
> From: Jason Wang [mailto:address@hidden
> Sent: Thursday, July 17, 2014 1:37 PM
> To: Wangkai (Kevin,C); Stefan Hajnoczi
> Cc: Lee yang; address@hidden; Stefan Hajnoczi;
> address@hidden; Michael S. Tsirkin
> Subject: Re: [Qemu-devel] [PATCH] Tap: fix vcpu long time io blocking
> on tap
> 
> On 07/17/2014 11:43 AM, Wangkai (Kevin,C) wrote:
> >
> >> -----Original Message-----
> >> From: Stefan Hajnoczi [mailto:address@hidden
> >> Sent: Tuesday, July 15, 2014 11:00 PM
> >> To: Wangkai (Kevin,C)
> >> Cc: Stefan Hajnoczi; Lee yang; address@hidden;
> >> address@hidden
> >> Subject: Re: [Qemu-devel] [PATCH] Tap: fix vcpu long time io
> blocking
> >> on tap
> >>
> >> On Mon, Jul 14, 2014 at 10:44:58AM +0000, Wangkai (Kevin,C) wrote:
> >>> Here the detail network:
> >>>
> >>> +--------------------------------------------+
> >>> | The host add tap1 and eth10 to bridge 'br1'|
> +-
> >> -------+
> >>> | +------------+                             |
> |
> >> send  |
> >>> | |   VM  eth1-+-tap1 --- bridge --- eth10 --+---------------------
> +
> >>> | | packets|
> >>> | +------------+                             |
> |
> >> |
> >>> +--------------------------------------------+
> >>> +--------------------------------------------+ +--------+
> >>>
> >>> Qemu start vm by virtio, use tap interface, option is:
> >>> -net nic,vlan=101, model=virtio -net
> >>> tap,vlan=101,ifname=tap1,script=no,downscript=no
> >> Use the newer -netdev/-device syntax to get offload support and
> >> slightly better performance:
> >>
> >> -netdev tap,id=tap0,ifname=tap1,script=no,downscript=no \ -device
> >> virtio-net-pci,netdev=tap0
> >>
> >>> And add tap1 and eth10 to bridge br1 in the host:
> >>> Brctl addif br1 tap1
> >>> Brctl addif br1 eth10
> >>>
> >>> total recv 505387 time 2000925 us:
> >>> means call tap_send once dealing 505387 packets, the packet payload
> >>> was 300 bytes, and time use for tap_send() was 2,000,925
> >>> micro-seconds, time was measured by record time stamp at function
> >> tap_send() start and end.
> >>> We just test the performance of VM.
> >> That is 150 MB of incoming packets in a single tap_send().  Network
> >> rx queues are maybe a few 1000 packets so I wonder what is going on
> here.
> >>
> >> Maybe more packets are arriving while QEMU is reading them and we
> >> keep looping.  That's strange though because the virtio-net rx
> >> virtqueue should fill up (it only has 256 entries).
> >>
> >> Can you investigate more and find out exactly what is going on?
> It's
> >> not clear yet that adding a budget is the solution or just hiding a
> >> deeper problem.
> >>
> >> Stefan
> > [Wangkai (Kevin,C)]
> >
> > Hi Stefan,
> >
> > I think I have found the problem, why 256 entries virtqueue cannot
> > prevent packets receiving.
> >
> > I have start one SMP guest, which have 2 cores, one core was pending
> > on Io, and the other core was receiving the packets, and QEMU filling
> > the virtqueue while the guest kernel was moving the packets out from
> > the queue and process.
> >
> > They were racing, only if the guest got enough packets and receive
> > slower than QEMU sending, the virtqueue full, then finish once
> receive.
> I hit the similar issue in the past. Using pktgen to inject packets
> from tap to guest directly, then guest was slow to respond any other io
> event.
> 
> This is similar to the tx and we have tx_burst or tx_timer to solve it.
> But for rx, probably we could do some limitation in tap_send() like
> this patch since e1000 may have the same issue.
> 
> Generically, we need some mechanism to guarantee the fairness between
> devices to prevent one from starving the others.
[Wangkai (Kevin,C)] 

Yes, the QEMU io shared one thread, and locked, fairness should be 
guaranteed between io events and the io lock should be as small as
possible.

> >
> > And I have tried -netdev/-device syntax start guest again, got very
> > few Improvement.
> >
> > Regards
> > Wangkai
> >