Re: [Qemu-discuss] GRO not happening in VM with VxLAN

[Vlad Yasevich]

On 07/01/2015 03:14 PM, Santosh R wrote:
> I am using 3.18 kernel.
> 
> #uname -a
> Linux Indra.asicdesigners.com 3.18.0 #11 SMP Wed Jul 1 21:49:44 IST 2015
> x86_64 x86_64 x86_64 GNU/Linux
> #cat /etc/issue
> Red Hat Enterprise Linux Server release 6.5 (Santiago)
> Kernel \r on an \m
> 
> As mentioned in my initial post, this is how I am starting the VM. How do
> we enable/pass tx-udp_tnl-segmentation feature in the below command? Is
> this an option while creating the VM?
> *# qemu-system-x86_64 -m 4096 -smp 4 -boot c -device
> virtio-net-pci,netdev=tap0,mac=00:11:22:33:44:55 -netdev
> tap,id=tap0,script=no,vhost=on -drive file=/root/vdisk_rhel65.img &*

You can't.  It's not supported yet.

-vlad

> 
> On Thu, Jul 2, 2015 at 12:37 AM, Vlad Yasevich <address@hidden> wrote:
> 
>> On 07/01/2015 02:50 PM, Santosh R wrote:
>>> Since the vxlan UDP header checksum is 0, udp_tunnel_gro_complete (called
>>> via vxlan_gro_complete) is setting SKB_GSO_UDP_TUNNEL in
>>> skb_shinfo(skb)->gso_type.
>>> Later when bridge interface tries to forward this packet to tap interface
>>> (br_dev_queue_push_xmit -> __dev_queue_xmit -> validate_xmit_skb)
>>> netif_needs_gso will succeed (skb_gso_ok returns 0)
>>> resulting in packet segmentation (which was earlier aggregated) as tap
>>> interface doesn't have this feature. Is there a way to enable
>>> tx-udp_tnl-segmentation for tap interface?
>>
>> No.  Those features are set by qemu when the guest is created and
>> mirror any features that are supported by the guest.  So, if the guest
>> virtio doesn't have that feature, neither will the tap device.
>>
>> Which kernel version are you using on the host?
>>
>> -vlad
>>
>>>
>>> # ethtool -K tap0 tx-udp_tnl-segmentation on
>>> Could not change any device features
>>>
>>> # ethtool -k tap0 | grep -i tnl
>>> tx-udp_tnl-segmentation: off [fixed]
>>>
>>> On Tue, Jun 30, 2015 at 12:24 PM, Santosh R <address@hidden>
>> wrote:
>>>
>>>>
>>>> On Mon, Jun 29, 2015 at 9:24 PM, Santosh R <address@hidden>
>> wrote:
>>>>
>>>>>
>>>>> On Mon, Jun 29, 2015 at 9:04 PM, Vlad Yasevich <address@hidden>
>>>>> wrote:
>>>>>
>>>>>> On 06/29/2015 01:46 AM, Santosh R wrote:
>>>>>>> All,
>>>>>>>
>>>>>>>    I am testing VxLAN performance in VM. For this I am using below
>>>>>> command
>>>>>>> to bring up the VM.
>>>>>>> # qemu-system-x86_64 -m 4096 -smp 4 -boot c -device
>>>>>>> virtio-net-pci,netdev=tap0,mac=00:11:22:33:44:55 -netdev
>>>>>>> tap,id=tap0,script=no,vhost=on  -drive file=/root/vdisk_rhel65.img &
>>>>>>>
>>>>>>> This is resulting in lower throughput in VM. On looking further I
>> found
>>>>>>> that GRO is not happing on the tap interface.
>>>>>>> Using tcpdump I could see GRO happeing in physcial nic, vxlan and
>>>>>> bridge
>>>>>>> interface. but _not_ on tap interface.
>>>>>>
>>>>>> I don't think this is GRO.  GRO  only happens at the lowest possible
>>>>>> layer.
>>>>>> For packets coming from external sources, that happens just above the
>> nic
>>>>>> when the packet enters the host for the first time.  From then on, the
>>>>>> features
>>>>>> of other devices determine if the large aggregated packet will be
>>>>>> segmented
>>>>>> again or not.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> So, I thought to use veth pair instead of the tap interface in VM.
>>>>>> What is
>>>>>>> the command to do this?
>>>>>>
>>>>>> veth pair will not help with this.  you need a tap or a macvtap to
>> have
>>>>>> VM  talk to host.
>>>>>>
>>>>>>> Also,
>>>>>>> 1. Why is rx-checksumming off for tap interface?
>>>>>>
>>>>>> because tap doesn't do checksum validation.  it allows lower level
>>>>>> devices or
>>>>>> VM to do so.
>>>>>>
>>>>>>> 2. Why is GRO not happening with tap interface?
>>>>>>>
>>>>>>
>>>>>> Again.  I don't think it's GRO.  Try capturing traffic at tap and see
>> if
>>>>>> you get
>>>>>> large packets (more the 1 mtu long) on the tap.  If not, then someone
>>>>>> else is fragmenting
>>>>>> packets and you need to find who.
>>>>>>
>>>>> [Santosh]:  Thanks for the reply.
>>>>> With VxLAN, aggregated packets are seen on physical, VxLAN and bridge
>>>>> interface but _not_ on the tap interface.
>>>>> Without VxLAN, I do see aggregated (GRO) packets all the way upto VM
>>>>> (i.e. physical, bridge and tap interface).
>>>>> As you rightly pointed, looks like VxLAN packets are getting
>> fragmented.
>>>>> Any pointers to look for?
>>>>>
>>>>>
>>>> By the way, I was referring below link for VxLAN testing and ran into
>> the
>>>> GRO issue.
>>>> This mentions "The veth0 is suppose to be connected to the VM, while the
>>>> veth1 is connected to the Linux bridge".
>>>> https://community.mellanox.com/docs/DOC-1473
>>>>
>>>
>>
>>
>