[lwip-users] Non-blocking netconn callback and TCP accept backlog

[Nick Brereton]

Hi,

I'm currently attempting to use lwIP's non-blocking netconn interface to 
implement an http based service. I am having trouble with new incoming 
connections, where if I delay accepting the connection from my listening 
connection, I seem to lose some of the data sent. I won't claim to be an expert 
at networking, or of lwIP so I may well have mis-configured something or have 
otherwise misunderstood how things should work. I would appreciate any hints or 
help that you can put my way about this problem.

To give a little background, I'm currently attempting to build a SOAP based 
server for a device that uses lwIP for networking (and, probably not relevant, 
but using FreeRTOS). I need to be able to handle multiple connection 
concurrently, some of which can take a significant amount of time to process; 
because of memory constraints I can only handle a small number of active 
connections at any one time (where small is less than the number of requests 
that could be outstanding). As alluded to, I have a listening connection (port 
80) that has been set to non-blocking mode, and I have a callback that is used 
to notify the main event loop of events that occur from the netconn events. I 
have configured the TCP_LISTEN_BACKLOG option.

When I get get a NETCONN_EVT_RECVPLUS event with 0 length for the listening 
connection I add this to a queue of accept event, which I then drain whenever I 
have memory available to process the connection. As I have understood it data 
should not start flowing until I have accepted the connection (presumably 
completion of the handshake should be delayed until  the netconn_accept call, 
thus preventing the client from sending any data?) and the connection should be 
sitting in the tcp listen queue.

What happens is that the connection is immediately accepted (presumably when 
the callback occurs?) and the client starts sending data immediately. But when 
I do accept the connection, there is never any data to retrieve with the 
netconn_recv call (no events & conn->recv_avail == 0) -- ie the data has 
effectively been lost. Of course at this point my server stops doing anything 
useful as it is expecting to receive data before it can do anything.

Poking through the lwip_stats structure, after I seem to have lost the data, 
the tcp stats reports that all of the data packets I can see with wireshark 
have been received without error (in this case recv=52, sent=42) and the memp 
stats tell me that there are no allocated buffers (i.e. there are no orphan 
packets that might contain my missing data). The tcp_active_pcb list contains 
established connections for all current and outstanding (backlogged) 
connections on my listening connection.

I have attached a wireshark pcap log of the network traffic (where the lwip 
server is at 192.168.7.1 and the client is 192.168.7.2); connections from 
client's port 57221, 57222, 57228, 57229 complete correctly; connections from 
client ports 57230 and 57231 have been netconn_accept'ed only after 57228 and 
57229 have been closed and think that they are waiting for the first data 
packet (i.e. a HTTP request line) and the connection from client port 57232 has 
never had netconn_accept called for it. As noted lwip_stats suggests that all 
of the packets have been accepted, but where there is no application data 
response packets the incoming data seems to have been 'lost' by lwIP, although 
no error is indicated.

My code looks something like this (callback & main loop); simplified of course 
but should give the key points:

void callback(struct netconn *pconn, enum netconn_evt event, u16_t len)
{
    switch(event) {
    case NETCONN_EVT_RCVPLUS:
        if(len > 0) {
            HTTPD_POST_MESSAGE(pconn, len, HTTPD_CMD_READRDY);
        } else if(pconn == httpd) {
            /* queue message noting that a connection is requested, will accept 
when ready */
        } /* otherwise this is caused by closing the socket ... */
        break;
    case NETCONN_EVT_RCVMINUS:  break; /* not useful? */
    case NETCONN_EVT_SENDPLUS:
        if(len > 0) /* otherwise doesn't seem to be useful */
            HTTPD_POST_MESSAGE(pconn, len, HTTPD_CMD_TXDONE);
        break;
    case NETCONN_EVT_SENDMINUS:  break; /* not useful? */
    case NETCONN_EVT_ERROR: HTTPD_POST_MESSAGE(pconn, len, HTTPD_CMD_ERROR); 
break;
    }
}

void main_loop(void* param)
{
    /* setup a non-blocking listening connection on port 80 */
    httpd = netconn_new_with_callback(NETCONN_TCP, callback);
    netconn_set_nonblocking(httpd, 1);
    netconn_bind(httpd, 0, HTTP_PORT);
    netconn_listen(httpd);

    while(1) {
        /* get events (network, timers etc) */
        switch(event) {
        case HTTPD_CMD_READRDY:
          netconn_recv(...);
          /* do stuff with data, free buffers etc.*/
          break;
        case HTTPD_CMD_READRDY:
          /* send more data on connection, or close */
          break;
        case HTTPD_CMD_TIMER:
            if(memory_available() && accept_pending()) {
                netconn_accept(httpd, &new_connection);
                /* associate new_state & new_connection, assume that we will 
get a NETCONN_EVT_RECVPLUS event to kick us off */
            }
        }
    }
}

My lwipopts.h file contains this:

#define BYTE_ORDER LITTLE_ENDIAN
#define TCPIP_THREAD_NAME                               "tcpip"
#define LWIP_HTTPD_MAX_TAG_NAME_LEN     20
#define LWIP_HTTPD_MAX_TAG_INSERT_LEN   1500
#define TCPIP_THREAD_PRIO                               (tskIDLE_PRIORITY + 1)
#define TCPIP_THREAD_STACKSIZE                  configMINIMAL_STACK_SIZE * 4
#define DEFAULT_TCP_RECVMBOX_SIZE               10
#define DEFAULT_ACCEPTMBOX_SIZE                 10
#define TCPIP_MBOX_SIZE                                 10
#define NO_SYS                                                  0
#define LWIP_SOCKET                             0
#define LWIP_NETCONN                            1
#define RECV_BUFSIZE_DEFAULT            1536
#define LWIP_SNMP                                               0
#define LWIP_IGMP                                               0
#define LWIP_ICMP                                               1
#define LWIP_DNS                                                0
#define LWIP_HAVE_LOOPIF                                0
#define TCP_LISTEN_BACKLOG                              1
#define LWIP_SO_RCVTIMEO                            1
#define LWIP_SO_RCVBUF                                  1
#define MEM_ALIGNMENT                           4
#define MEM_SIZE                                        10240
#define MEMP_NUM_PBUF                           20
#define LWIP_RAW                                        0
#define MEMP_NUM_RAW_PCB                        0
#define MEMP_NUM_UDP_PCB                        2
#define MEMP_NUM_TCP_PCB                        40
#define MEMP_NUM_TCP_PCB_LISTEN         2
#define MEMP_NUM_TCP_SEG                        20
#define DEFAULT_UDP_RECVMBOX_SIZE       10
#define MEMP_NUM_SYS_TIMEOUT            15
#define MEMP_NUM_NETBUF                 10
#define MEMP_NUM_NETCONN                10
#define MEMP_NUM_TCPIP_MSG_API          6
#define MEMP_NUM_TCPIP_MSG_INPKT        6
#define MEMP_NUM_ARP_QUEUE                      5
#define PBUF_POOL_SIZE                          10
#define PBUF_POOL_BUFSIZE                       375
#define PBUF_LINK_HLEN                          16
#define SYS_LIGHTWEIGHT_PROT            (NO_SYS==0)
#define LWIP_TCP                                        1
#define TCP_TTL                                         255
#define TCP_QUEUE_OOSEQ                         1
#define TCP_MSS                                         1460
#define TCP_SND_BUF                                     ( TCP_MSS * 2 )
#define TCP_SND_QUEUELEN                        (4 * TCP_SND_BUF/TCP_MSS)
#define TCP_SNDLOWAT                            (TCP_SND_BUF/2)
#define TCP_WND                                         ( PBUF_POOL_SIZE * 
(PBUF_POOL_BUFSIZE - (PBUF_LINK_HLEN + PBUF_IP_HLEN + PBUF_TRANSPORT_HLEN)))
#define TCP_MAXRTX                                      12
#define TCP_SYNMAXRTX                           4
#define LWIP_ARP                                        1
#define ARP_TABLE_SIZE                          10
#define ARP_QUEUEING                            1
#define IP_FORWARD                                      0
#define IP_REASSEMBLY                           0
#define IP_REASS_MAX_PBUFS                      10
#define MEMP_NUM_REASSDATA                      10
#define IP_FRAG                                         0
#define ICMP_TTL                                        255
#define LWIP_DHCP                                       0
#define DHCP_DOES_ARP_CHECK                     (LWIP_DHCP)
#define LWIP_AUTOIP                                     0
#define LWIP_DHCP_AUTOIP_COOP           (LWIP_DHCP && LWIP_AUTOIP)
#define LWIP_UDP                                        1
#define LWIP_UDPLITE                            1
#define UDP_TTL                                         255
#define LWIP_STATS                                      1
#define LWIP_STATS_DISPLAY                      0
#define PPP_SUPPORT                                 0

I can't say that I have fully understood how to configure lwIP, so this is 
lightly modified from a FreeRTOS example and may not be quite what I need but 
it seems to have worked so far.

I should also note that when I have tried using a single loop with a blocking 
server I have not lost data, although a simple loop & blocking netconns is not 
practical for my use-case.

Many thanks for any help,

Nick

httpd_accept_breaks.pcap
Description: httpd_accept_breaks.pcap