Re: [Qemu-devel] tap devices not receiving packets from a bridge

[Peter Lieven]

Am 23.11.2012 um 12:01 schrieb Michael S. Tsirkin:

> On Fri, Nov 23, 2012 at 10:41:21AM +0100, Peter Lieven wrote:
>> 
>> Am 23.11.2012 um 08:02 schrieb Stefan Hajnoczi:
>> 
>>> On Thu, Nov 22, 2012 at 03:29:52PM +0100, Peter Lieven wrote:
>>>> is anyone aware of a problem with the linux network bridge that in very 
>>>> rare circumstances stops
>>>> a bridge from sending pakets to a tap device?
>>>> 
>>>> My problem occurs in conjunction with vanilla qemu-kvm-1.2.0 and Ubuntu 
>>>> Kernel 3.2.0-34.53
>>>> which is based on Linux 3.2.33.
>>>> 
>>>> I was not yet able to reproduce the issue, it happens in really rare 
>>>> cases. The symptom is that
>>>> the tap does not have any TX packets. RX is working fine. I see the 
>>>> packets coming in at
>>>> the physical interface on the host, but they are not forwarded to the tap 
>>>> interface.
>>>> The bridge itself has learnt the mac address of the vServer that is 
>>>> connected to the tap interface.
>>>> It does not help to toggle the bridge link status,  the tap interface 
>>>> status or the interface in the vServer.
>>>> It seems that problem occurs if a tap interface that has previously been 
>>>> used, but set to nonpersistent
>>>> is set persistent again and then is by chance assigned to the same vServer 
>>>> (=same mac address on same
>>>> bridge) again. Unfortunately it seems not to be reproducible.
>>> 
>>> Not sure but this patch from Michael Tsirkin may help - it solves an
>>> issue with persistent tap devices:
>>> 
>>> http://patchwork.ozlabs.org/patch/198598/
>> 
>> Hi Stefan,
>> 
>> thanks for the pointer. I have seen this patch, but I have neglected it 
>> because it was dealing
>> with persistent taps. But maybe the taps in the kernel are not deleted 
>> directly. 
>> Can you remember what the syptomps of the above issue have been? Sorry for
>> being vague, but I currently have no clue whats going on.
>> 
>> Can someone who has more internal knowledge of the bridging/tap code say if 
>> qemu can
>> be responsible at all if the tap device is not receiving packets from the 
>> bridge.
>> 
>> If I have the following config. Lets say packets coming in via physical 
>> interface eth1.123,
>> and a bridge called br123.I further have a virtual machine with tap0. Both 
>> eth1.123
>> and tap0 are member of br123. 
>> 
>> If the issue occurs the vServer has no network connectivity inbound. If I 
>> sent a ping
>> from the vServer I see it on tap0 and leaving on eth1.123. I see further the 
>> arp reply coming
>> in via eth1.123, but the reply can't be seen on tap0.
>> 
>> Peter
> 
> If guest is not consuming packets, a TX queue in tap device
> will with time overrun (there's space for 1000 packets there).
> This is code from tun:

From what I remember there where zero TX packets and no TX errors
on the device.

Might it be that this queue is somehow not cleared correctly when
the device is reassigned (although it was nonpersistant in between).

Thank you,
Peter

> 
>        if (skb_queue_len(&tfile->socket.sk->sk_receive_queue)
>> = dev->tx_queue_len / tun->numqueues){
>                if (!(tun->flags & TUN_ONE_QUEUE)) {
>                        /* Normal queueing mode. */
>                        /* Packet scheduler handles dropping of further
> * packets. */
>                        netif_stop_subqueue(dev, txq);
> 
>                        /* We won't see all dropped packets
> * individually, so overrun
>                         * error is more appropriate. */
>                        dev->stats.tx_fifo_errors++;
> 
> 
> So you can detect that this triggered by looking at fifo errors counter in 
> device.
> 
> Once this happens TX queue is stopped, then you hit this path:
> 
>                        if (!netif_xmit_stopped(txq)) {
>                                __this_cpu_inc(xmit_recursion);
>                                rc = dev_hard_start_xmit(skb, dev, txq);
>                                __this_cpu_dec(xmit_recursion);
>                                if (dev_xmit_complete(rc)) {
>                                        HARD_TX_UNLOCK(dev, txq);
>                                        goto out;
>                                }
>                        }
> 
> so packets are not passed to device anymore.
> It will stay this way until guest consumes some packets and
> queue is restarted.
> 
>>> 
>>> Stefan