Re: [lwip-users] Out of sequence packets lost between TCP and HTTP server

[Johan Borkhuis]

Sorry about the user-name in the previous message, did not yet setup my
email client correctly. That should be fixed now.

On Fri, Feb 23, 2018 at 11:06:16AM +0100, Maillist wrote:
> Hello,
> 
> I am running into some issues with the httpd application in LwIP, which looks
> like an issue between TCP receive and HTTP server.
> I am trying to send a file to my target, but I am only receiving part of it in
> my application. I implemented the httpd_post_begin, httpd_post_receive_data 
> and
> httpd_post_finished in my application, but I a missing part of the data,
> although the client thinks all data was transmitted (which is to be expected, 
> as
> HTTP is a TCP stream).
> 
> My configuration:
> - LwIP 2.0.3 and FreeRTOS 10.0.1
> - TCP_MSS = 1460
> - TCP_WND = (8 * TCP_MSS)
> - TCP_QUEUE_OOSEQ = 1
> - Client: Windows 10 with Chrome or Firefox
> - Target: NXP ARM M4 platform
> 
> I am missing data with the size of one TCP_WND-1, so 7 frames. The data was
> transmitted over the line, but at one moment the internal buffers are full 
> and I
> start seeing duplicate ACKs. The data referred to in the duplicate ACK is
> retransmitted by the client.
> Then, after some time, I see another ACK (forward ACK to the next data frame)
> and the client continues sending the data, as it assumes that all data
> inbetween was received and processed.
> 
> The detailed trace, sending is from client (I have the complete Wireshark log,
> if someone is interested):
> 
> <--> SYM/ACK sequence
>  --> Send POST request
>  --> Send data frame 1
>  --> Send data frame 2
>  --> Send data frame 3
> <--  ACK frame 2
>  --> Send data frame 4
>  --> Send data frame 5
>  --> Send data frame 6
> <--  ACK frame 4
>  --> Send data frame 7
>  --> Send data frame 8
>  --> Send data frame 9
>  --> Send data frame 10
> <--  ACK frame 6
>  --> Send data frame 11
>  --> Send data frame 12
>  --> Send data frame 13 (Window full)
> <--  ACK frame 8
>  --> Send data frame 14
>  --> Send data frame 15 (Window full)
> <--  ACK frame 10
>  --> Send data frame 16
>  --> Send data frame 17 (Window full)
> <--  ACK frame 12
>  --> Send data frame 18
>  --> Send data frame 19 (Window full)
> <--  ACK frame 13
>  --> Send data frame 20 (Window full)
> <--  ACK frame 13 (Dup)
> <--  ACK frame 13 (Dup)
> <--  ACK frame 13 (Dup)
>  --> Retransmit data frame 13
> <--  ACK frame 13 (Dup)
> <--  ACK frame 13 (Dup)
> <--  ACK frame 13 (Dup)
> <--  ACK frame 21 (Forward)
>  --> Send data frame 21
>  --> Send data frame 22
>  --> Send data frame 23
>  --> Send data frame 24
> <--  ACK frame 23
>  --> Send data frame 25
>  --> Send data frame 26
> etc.
> 
> In the above scenario I am missing frames 14 through 20, all other data is
> received. 
> 
> Below are the sequence numbers as seen on TCP level (function tcp_input,
> variable seqno) and on HTTP level (function http_recv, variable pcb->snd_wl1)
> from packet 10 until 23 as seen by my application (the packets can be
> distinguished as I created a file with section of 1460 bytes with increasing
> numbers in it). The shown order is the order of the processing, so first TCP
> level, then HTTPD level and then application.
> 
> TCP 3852526646
> HTTPD 3852526646
> Packet 10
> 
> TCP 3852528106
> HTTPD 3852528106
> Packet 11
> 
> TCP 3852529566
> HTTPD 3852529566
> Packet 12
> 
> TCP 3852532486  (packet 14)
> TCP 3852533946  (packet 15)
> TCP 3852535406  (packet 16)
> TCP 3852536866  (packet 17)
> TCP 3852538326  (packet 18)
> TCP 3852539786  (packet 19)
> TCP 3852541246  (packet 20)
> TCP 3852531026  (packet 13)
> HTTPD 3852541246
> Packet 13
> 
> TCP 3852542706
> HTTPD 3852542706
> Packet 21
> 
> TCP 3852544166
> HTTPD 3852544166
> Packet 22
> 
> TCP 3852545626
> HTTPD 3852545626
> Packet 23
> 
> Until packet 12 and from packet 21 the TCP and HTTP sequence number can be
> directly traced back to the packet number.
> 
> After packet 12 I would expect sequence number 3852531026, but this packets
> was not seen by the TCP stack (might have been dropped in hardware). Then,
> after the DUP ACKs packet 13 is received and processed, but the other packets
> that were already received are not processed.
> 
> The sequence number seen by the HTTP server is the number from packet 20, but
> this could be because I used pcb->snd_wl1 to determine the sequence number in
> the HTTP server.
> 
> Why are the packets between 13 and 20 not received by the HTTP server? Do I
> need to set another callback to process the out-of-sequence packets or is
> there another option that can be set to make sure the packets are passed to
> the application?
> 
> When I change TCP_QUEUE_OOSEQ to 0 the problem disappears, but that also means
> that a number of frames are retransmitted, which I would like to avoid. I also
> played with the TCP window size, but that does not get rid of the problem, the
> issue only appears later when I use a larger window size.
> 
> Thanks for your answer and advice.
> 
> Regards,
> Johan Borkhuis
> 
> _______________________________________________
> lwip-users mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/lwip-users