[lwip-users] TCP retransmission flooding at end of stream

[Michael Steinecke]

Hello Folks,

currently I'm struggling while creating an application for a custom
STM32F429ZG based custom board using LwIP.
I need to achieve a throughput of at least 2 MBit/s, according our
requirements. Due to the HW this should be possible.

The application is based on the STMCubeMX V4.3.0 and FW Library 1.3.0.
However I've written a zero-copy EthernetIF driver in RX&TX.

For the application, the MCU based device acts as a TCP/IP server with our
own application protocol, implemented using the RAW TCP API.
The implementation is inspired by the HTTP Server example. Also the
guidelines regarding throughput from the LwIP wiki has been followed.
The FW Library bug in the Ethernet IRQ, eating fast packets is fixed. I'm
using custom memory pools. there is most likely no memory issue.
Also all the priorities are ok.

To achieve maximum throughput, I have done some further, potential dangerous
changes:
- The MSS, TCP_SND_BUF, pbuf len, etc is increased from u16_t to u32_t. The
MTU is still 1500. The problem with u16_t is the limitation the send buffer
to a maximum of 0xFFFE bytes. For some checks in LwIP 0xFFFF is interpreted
as 0x0000, but is need to have segments of 0xFFFF bytes for a fast
processing of SD Card pages.
- I decreased the TCP timer intervals from 250 ms to 10 ms. A even higher
rate tends to produce a lot of retransmissions.

Currently I can achieve almost the desired speed for one file transfer.
However, at the end of the transfer, there is one strange bug. I couldn't
find a similar case in the forums, so far.
Before the last frame of the last segment is sent, LwIP starts to retransmit
a bunch of TCP packets. The actual number is variable, but usually somewhere
between 10 and 25 frames.

Interestingly there is always this pattern:
MCU -> PC Packet 1460 Bytes [ACK]
MCU -> PC Packet 1460 Bytes [ACK]
PC -> MCU ACK on both, 0 Bytes
MCU -> PC Packet 1460 Bytes [ACK]
MCU -> PC Packet 1460 Bytes [ACK]
PC -> MCU ACK on both, 0 Bytes
...
MCU -> PC Packet 1460 Bytes [ACK]
MCU -> PC Packet 1460 Bytes [ACK, PSH]
PC -> MCU ACK on both, 0 Bytes
............


Then the following occurs on the last segment:
(for this example, the Seq is relative to the begin of the segment)
MCU -> PC Packet 1460 Bytes [ACK] [Seq: 0000]
MCU -> PC Packet 1460 Bytes [ACK] [Seq: 1460]
PC -> MCU ACK on both, 0 Bytes [Ack: 2920]
MCU -> PC Packet 1460 Bytes [ACK] [Seq: 2920]
MCU -> PC Packet 1460 Bytes [ACK] [Seq: 4380]
PC -> MCU ACK on both, 0 Bytes [Ack: 4380]
...
MCU -> PC Packet 1460 Bytes [TCP Retransmission] [ACK] [Seq: 0000]
PC -> MCU 0 Bytes [TCP DUP ACK] [Ack: 4380]
MCU -> PC Packet 1460 Bytes [TCP Retransmission] [ACK] [Seq: 2920] (Packet
with Seq 1460 is NOT retransmitted!)
PC -> MCU 0 Bytes [TCP DUP ACK] [Ack: 4380]
MCU -> PC Packet 1460 Bytes [TCP Retransmission] [ACK] [Seq: 4380]
PC -> MCU 0 Bytes [TCP DUP ACK] [Ack: 4380]
...
MCU -> PC Packet 1198 Bytes [ACK, PSH] [Seq: 65700]
PC -> MCU ACK on last one, sending new command, 38 Bytes [ACK, PSH] [Seq:
66898]
MCU -> PC Packet 1198 Bytes [TCP Retransmission] [ACK, PSH] [Seq: 65700]
PC -> MCU 0 Bytes [TCP DUP ACK #1] [ACK, PSH] [Seq: 66898]
MCU -> PC Packet 1198 Bytes [TCP Retransmission] [ACK, PSH] [Seq: 65700]
PC -> MCU0 Bytes [TCP DUP ACK #2] [ACK, PSH] [Seq: 66898]
MCU -> PC Packet 1198 Bytes [TCP Retransmission] [ACK, PSH] [Seq: 65700]
PC -> MCU 0 Bytes [TCP DUP ACK #3] [ACK, PSH] [Seq: 66898]
PC -> MCU ACK on last one, sending new command, 38 Bytes [TCP
Retransmission] [ACK, PSH] [Seq: 66898]
...

Afterwards, nearly every packet is transmitted at least twice in both
directions. Effectively no data transfer possible anymore.
In the pcap file, 192.168.111.200 is MCU and .63 the PC. The interesting
part is starting with frame 2056, the first retransmitted frame. 2057 is not
retransmitted - that means frame 2058 must have been processed by LwIP -
right?

The data is passed to LwIP by tcp_write().
One segment consists of two writes:
tcp_write(pcb, [ptr to 8 bytes of header], 8, TCP_WRITE_FLAG_COPY |
TCP_WRITE_FLAG_MORE)
tcp_write(pcb, [ptr to up to 65535 bytes of data], len, 0) // the data is
stored in elements of the memory pool and freed by the tcp_sent callback

According to lwip_stats there is no memory leak and no packet drop. Has
someone seen something similar? Any clue?
There are some other scenarios, leading to similar behaviors. 


SD_download_breaks.pcap
<http://lwip.100.n7.nabble.com/file/n23275/SD_download_breaks.pcap>  





--
View this message in context: 
http://lwip.100.n7.nabble.com/TCP-retransmission-flooding-at-end-of-stream-tp23275.html
Sent from the lwip-users mailing list archive at Nabble.com.