Re: [Bug 1889943] Improper TCP/IP packet splitting on e1000e/vmxnet3

[Yan Vugenfirer]

Hello Patrick,

If you are using  QEMU version 4.2, then it is missing recent patches fixing 
IPv6 and TSO behaviour:
https://www.mail-archive.com/qemu-devel@nongnu.org/msg723411.html
https://www.mail-archive.com/qemu-devel@nongnu.org/msg723412.html

Can you check that the above patches solve your issues?


Best regards,
Yan.

> On 2 Aug 2020, at 6:59 PM, Patrick Magauran <1889943@bugs.launchpad.net> 
> wrote:
> 
> Some more clarifications:
> It appears the QEMU does turn on the vnet_hdr flag of the tap interface in 
> most cases, not just host-only networks. My previous assumption was due to 
> the way the libvirt manages it, only setting it if the virtio interface is 
> used.
> 
> Still, for software fragmentation implementations, ip fragmentation
> should be a last resort.
> 
> I have also confirmed a suspicion that the current implementation of sw
> fragmentation will not work with IPV6. It creates malformed packets as
> ipv6 requires a different setup of headers to fragment. Thanks to the
> many redundancies in the network stack, the packets eventually arrive at
> the host server correctly formed, but we should not rely on this fact.
> 
> ** Description changed:
> 
> + Update: The sw implementation of fragmentation also creates malformed
> + IPv6 packets when their size is above the MTU. See comment #3
> + 
>  Problem Description:
> - When using a tap interface and the guest sends a TCP packet that would need 
> to be segmented, it is fragmented using IP fragmentation. The host does not 
> reassemble the IP fragments and forwards them to the next hop. This causes 
> issues on certain ISPs, which seemingly reject IP fragments(Verizon Fios). 
> - This issue occurs on the e1000e and vmxnet3 NIC models, and possibly 
> others. It does not occur on the virtio(which passes the entire packet 
> through to the host w/o fragmentation or segmentation) or the e1000 model(). 
> + When using a tap interface and the guest sends a TCP packet that would need 
> to be segmented, it is fragmented using IP fragmentation. The host does not 
> reassemble the IP fragments and forwards them to the next hop. This causes 
> issues on certain ISPs, which seemingly reject IP fragments(Verizon Fios).
> + This issue occurs on the e1000e and vmxnet3 NIC models, and possibly 
> others. It does not occur on the virtio(which passes the entire packet 
> through to the host w/o fragmentation or segmentation) or the e1000 model().
> 
>  Test scenario:
>  Setup a tap and network bridge using the directions here: 
> https://gist.github.com/extremecoders-re/e8fd8a67a515fee0c873dcafc81d811c
>  Boot the machine into any modern guest(a Fedora 31 live iso was used for 
> testing)
>  Begin a wireshark capture on the host machine
>  On the host(or another machine on the network) run: npx http-echo-server(See 
> https://github.com/watson/http-echo-server)
>  On the guest run
>  Curl -d “Lorem ipsum dolor sit amet, consectetur adipiscing elit. Maecenas 
> venenatis viverra ipsum, ac tincidunt est rhoncus eu. Suspendisse vehicula 
> congue ante, non rhoncus elit tempus vitae. Duis ac leo massa. Donec rutrum 
> condimentum turpis nec ultricies. Duis laoreet elit eu arcu pulvinar, vitae 
> congue neque mattis. Mauris sed ante nunc. Vestibulum vitae urna a tellus 
> maximus sagittis. Vivamus luctus pellentesque neque, vel tempor purus porta 
> ut. Phasellus at quam bibendum, fermentum libero sit amet, ullamcorper 
> mauris. In rutrum sit amet dui id maximus. Ut lectus ligula, hendrerit nec 
> aliquam non, finibus a turpis. Proin scelerisque convallis ante, et pharetra 
> elit. Donec nunc nisl, viverra vitae dui at, posuere rhoncus nibh. Mauris in 
> massa quis neque posuere placerat quis quis massa. Donec quis lacus ligula. 
> Donec mollis vel nisi eget elementum. Nam id magna porta nunc consectetur 
> efficitur ac quis lorem. Cras faucibus vel ex porttitor mattis. Praesent in 
> mattis tortor. In venenatis convallis quam, in posuere nibh. Proin non 
> dignissim massa. Cras at mi ut lorem tristique fringilla. Nulla ac quam 
> condimentum metus tincidunt vulputate ut at leo. Nunc pellentesque, nunc vel 
> rhoncus condimentum, arcu sem molestie augue, in suscipit mauris odio mollis 
> odio. Integer hendrerit lectus a leo facilisis, in accumsan urna maximus. Nam 
> nec odio volutpat, varius est id, tempus libero. Vestibulum lobortis tortor 
> quam, ac scelerisque urna rhoncus in. Etiam tempor, est sit amet vulputate 
> molestie, urna neque sodales leo, sit amet blandit risus felis sed est. Nulla 
> eu eros nec tortor dapibus maximus faucibus ut erat. Ut pharetra tempor massa 
> in bibendum. Interdum et malesuada fames ac ante ipsum primis in faucibus. 
> Etiam mattis molestie felis eu efficitur. Morbi tincidunt consectetur diam 
> tincidunt feugiat. Morbi euismod ut lorem finibus pellentesque. Aliquam eu 
> porta ex. Aliquam cursus, orci sit amet volutpat egestas, est est pulvinar 
> erat, sed luctus nisl ligula eget justo vestibulum.” <ECHOSERVERIP:PORT>
> 
>  2000 bytes of Lorem Ipsum taken from https://www.lipsum.com/
> 
>  Compare results from an e1000, a virtio, and a e1000e card:
>  +--------+-----------+---------+------------+
>  | Model  | Fragment  | Segment | Wire Size  |
>  +--------+-----------+---------+------------+
>  | e1000e | Yes       | NO      | 1484 + 621 |
>  +--------+-----------+---------+------------+
>  | e1000  | No        | Yes     | 1516 + 620 |
>  +--------+-----------+---------+------------+
>  | Virtio | NO        | NO      | 2068       |
>  +--------+-----------+---------+------------+
> 
>  Expected Results:
>  TCP Segment to proper size OR pass full size to host and let the host split 
> if necessary.
> 
>  Configuration changes that did not work:
>  Disable host, guest, router firewalls
>  Different Hosts
>  Different Physical NICs
>  Libvirt based NAT/Routed modes
>  Fedora 32 vs 31
>  Qemu 4.2.0 vs github commit d74824cf7c8b352f9045e949dc636c7207a41eee
> 
>  System Information:
>  lsb_release -rd
>  Description: Fedora release 32 (Thirty Two)
>  Release:     32
> 
>  uname -a
>  Linux pats-laptop-linux 5.7.10-201.fc32.x86_64 #1 SMP Thu Jul 23 00:58:39 
> UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
> 
>  I can provide additional logs, debug info, etc. if needed.
> 
> -- 
> You received this bug notification because you are a member of qemu-
> devel-ml, which is subscribed to QEMU.
> https://bugs.launchpad.net/bugs/1889943
> 
> Title:
>  Improper TCP/IP packet splitting on e1000e/vmxnet3
> 
> Status in QEMU:
>  New
> 
> Bug description:
>  Update: The sw implementation of fragmentation also creates malformed
>  IPv6 packets when their size is above the MTU. See comment #3
> 
>  Problem Description:
>  When using a tap interface and the guest sends a TCP packet that would need 
> to be segmented, it is fragmented using IP fragmentation. The host does not 
> reassemble the IP fragments and forwards them to the next hop. This causes 
> issues on certain ISPs, which seemingly reject IP fragments(Verizon Fios).
>  This issue occurs on the e1000e and vmxnet3 NIC models, and possibly others. 
> It does not occur on the virtio(which passes the entire packet through to the 
> host w/o fragmentation or segmentation) or the e1000 model().
> 
>  Test scenario:
>  Setup a tap and network bridge using the directions here: 
> https://gist.github.com/extremecoders-re/e8fd8a67a515fee0c873dcafc81d811c
>  Boot the machine into any modern guest(a Fedora 31 live iso was used for 
> testing)
>  Begin a wireshark capture on the host machine
>  On the host(or another machine on the network) run: npx http-echo-server(See 
> https://github.com/watson/http-echo-server)
>  On the guest run
>  Curl -d “Lorem ipsum dolor sit amet, consectetur adipiscing elit. Maecenas 
> venenatis viverra ipsum, ac tincidunt est rhoncus eu. Suspendisse vehicula 
> congue ante, non rhoncus elit tempus vitae. Duis ac leo massa. Donec rutrum 
> condimentum turpis nec ultricies. Duis laoreet elit eu arcu pulvinar, vitae 
> congue neque mattis. Mauris sed ante nunc. Vestibulum vitae urna a tellus 
> maximus sagittis. Vivamus luctus pellentesque neque, vel tempor purus porta 
> ut. Phasellus at quam bibendum, fermentum libero sit amet, ullamcorper 
> mauris. In rutrum sit amet dui id maximus. Ut lectus ligula, hendrerit nec 
> aliquam non, finibus a turpis. Proin scelerisque convallis ante, et pharetra 
> elit. Donec nunc nisl, viverra vitae dui at, posuere rhoncus nibh. Mauris in 
> massa quis neque posuere placerat quis quis massa. Donec quis lacus ligula. 
> Donec mollis vel nisi eget elementum. Nam id magna porta nunc consectetur 
> efficitur ac quis lorem. Cras faucibus vel ex porttitor mattis. Praesent in 
> mattis tortor. In venenatis convallis quam, in posuere nibh. Proin non 
> dignissim massa. Cras at mi ut lorem tristique fringilla. Nulla ac quam 
> condimentum metus tincidunt vulputate ut at leo. Nunc pellentesque, nunc vel 
> rhoncus condimentum, arcu sem molestie augue, in suscipit mauris odio mollis 
> odio. Integer hendrerit lectus a leo facilisis, in accumsan urna maximus. Nam 
> nec odio volutpat, varius est id, tempus libero. Vestibulum lobortis tortor 
> quam, ac scelerisque urna rhoncus in. Etiam tempor, est sit amet vulputate 
> molestie, urna neque sodales leo, sit amet blandit risus felis sed est. Nulla 
> eu eros nec tortor dapibus maximus faucibus ut erat. Ut pharetra tempor massa 
> in bibendum. Interdum et malesuada fames ac ante ipsum primis in faucibus. 
> Etiam mattis molestie felis eu efficitur. Morbi tincidunt consectetur diam 
> tincidunt feugiat. Morbi euismod ut lorem finibus pellentesque. Aliquam eu 
> porta ex. Aliquam cursus, orci sit amet volutpat egestas, est est pulvinar 
> erat, sed luctus nisl ligula eget justo vestibulum.” <ECHOSERVERIP:PORT>
> 
>  2000 bytes of Lorem Ipsum taken from https://www.lipsum.com/
> 
>  Compare results from an e1000, a virtio, and a e1000e card:
>  +--------+-----------+---------+------------+
>  | Model  | Fragment  | Segment | Wire Size  |
>  +--------+-----------+---------+------------+
>  | e1000e | Yes       | NO      | 1484 + 621 |
>  +--------+-----------+---------+------------+
>  | e1000  | No        | Yes     | 1516 + 620 |
>  +--------+-----------+---------+------------+
>  | Virtio | NO        | NO      | 2068       |
>  +--------+-----------+---------+------------+
> 
>  Expected Results:
>  TCP Segment to proper size OR pass full size to host and let the host split 
> if necessary.
> 
>  Configuration changes that did not work:
>  Disable host, guest, router firewalls
>  Different Hosts
>  Different Physical NICs
>  Libvirt based NAT/Routed modes
>  Fedora 32 vs 31
>  Qemu 4.2.0 vs github commit d74824cf7c8b352f9045e949dc636c7207a41eee
> 
>  System Information:
>  lsb_release -rd
>  Description: Fedora release 32 (Thirty Two)
>  Release:     32
> 
>  uname -a
>  Linux pats-laptop-linux 5.7.10-201.fc32.x86_64 #1 SMP Thu Jul 23 00:58:39 
> UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
> 
>  I can provide additional logs, debug info, etc. if needed.
> 
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/qemu/+bug/1889943/+subscriptions
>