[lwip-users] Out of sequence packets lost between TCP and HTTP server

[Maillist]

Hello,

I am running into some issues with the httpd application in LwIP, which looks
like an issue between TCP receive and HTTP server.
I am trying to send a file to my target, but I am only receiving part of it in
my application. I implemented the httpd_post_begin, httpd_post_receive_data and
httpd_post_finished in my application, but I a missing part of the data,
although the client thinks all data was transmitted (which is to be expected, as
HTTP is a TCP stream).

My configuration:
- LwIP 2.0.3 and FreeRTOS 10.0.1
- TCP_MSS = 1460
- TCP_WND = (8 * TCP_MSS)
- TCP_QUEUE_OOSEQ = 1
- Client: Windows 10 with Chrome or Firefox
- Target: NXP ARM M4 platform

I am missing data with the size of one TCP_WND-1, so 7 frames. The data was
transmitted over the line, but at one moment the internal buffers are full and I
start seeing duplicate ACKs. The data referred to in the duplicate ACK is
retransmitted by the client.
Then, after some time, I see another ACK (forward ACK to the next data frame)
and the client continues sending the data, as it assumes that all data
inbetween was received and processed.

The detailed trace, sending is from client (I have the complete Wireshark log,
if someone is interested):

<--> SYM/ACK sequence
 --> Send POST request
 --> Send data frame 1
 --> Send data frame 2
 --> Send data frame 3
<--  ACK frame 2
 --> Send data frame 4
 --> Send data frame 5
 --> Send data frame 6
<--  ACK frame 4
 --> Send data frame 7
 --> Send data frame 8
 --> Send data frame 9
 --> Send data frame 10
<--  ACK frame 6
 --> Send data frame 11
 --> Send data frame 12
 --> Send data frame 13 (Window full)
<--  ACK frame 8
 --> Send data frame 14
 --> Send data frame 15 (Window full)
<--  ACK frame 10
 --> Send data frame 16
 --> Send data frame 17 (Window full)
<--  ACK frame 12
 --> Send data frame 18
 --> Send data frame 19 (Window full)
<--  ACK frame 13
 --> Send data frame 20 (Window full)
<--  ACK frame 13 (Dup)
<--  ACK frame 13 (Dup)
<--  ACK frame 13 (Dup)
 --> Retransmit data frame 13
<--  ACK frame 13 (Dup)
<--  ACK frame 13 (Dup)
<--  ACK frame 13 (Dup)
<--  ACK frame 21 (Forward)
 --> Send data frame 21
 --> Send data frame 22
 --> Send data frame 23
 --> Send data frame 24
<--  ACK frame 23
 --> Send data frame 25
 --> Send data frame 26
etc.

In the above scenario I am missing frames 14 through 20, all other data is
received. 

Below are the sequence numbers as seen on TCP level (function tcp_input,
variable seqno) and on HTTP level (function http_recv, variable pcb->snd_wl1)
from packet 10 until 23 as seen by my application (the packets can be
distinguished as I created a file with section of 1460 bytes with increasing
numbers in it). The shown order is the order of the processing, so first TCP
level, then HTTPD level and then application.

TCP 3852526646
HTTPD 3852526646
Packet 10

TCP 3852528106
HTTPD 3852528106
Packet 11

TCP 3852529566
HTTPD 3852529566
Packet 12

TCP 3852532486  (packet 14)
TCP 3852533946  (packet 15)
TCP 3852535406  (packet 16)
TCP 3852536866  (packet 17)
TCP 3852538326  (packet 18)
TCP 3852539786  (packet 19)
TCP 3852541246  (packet 20)
TCP 3852531026  (packet 13)
HTTPD 3852541246
Packet 13

TCP 3852542706
HTTPD 3852542706
Packet 21

TCP 3852544166
HTTPD 3852544166
Packet 22

TCP 3852545626
HTTPD 3852545626
Packet 23

Until packet 12 and from packet 21 the TCP and HTTP sequence number can be
directly traced back to the packet number.

After packet 12 I would expect sequence number 3852531026, but this packets
was not seen by the TCP stack (might have been dropped in hardware). Then,
after the DUP ACKs packet 13 is received and processed, but the other packets
that were already received are not processed.

The sequence number seen by the HTTP server is the number from packet 20, but
this could be because I used pcb->snd_wl1 to determine the sequence number in
the HTTP server.

Why are the packets between 13 and 20 not received by the HTTP server? Do I
need to set another callback to process the out-of-sequence packets or is
there another option that can be set to make sure the packets are passed to
the application?

When I change TCP_QUEUE_OOSEQ to 0 the problem disappears, but that also means
that a number of frames are retransmitted, which I would like to avoid. I also
played with the TCP window size, but that does not get rid of the problem, the
issue only appears later when I use a larger window size.

Thanks for your answer and advice.

Regards,
Johan Borkhuis