[lwip-users] TCP speed drops to a crawl.

[Michael Williamson]

Hi,

  I am fairly new to lwip.  We are using lwip 1.1.0 with a port to TI's 
Code Composer BIOS for a 6711 processor.  The stack seems to be working, 
but we are seeing a strange behaviour with a simple TCP packet 
throughput test that smells like a configuration problem.  We are using 
the BSD sockets.c style API and have code that looks something like that 
below. 

If we set TCP_NODELAY, we get about 3100 packets very quickly (about 
1000 per second) then the rate drops substantially and it looks like 
(using ethereal) the lwip stack is waiting for an ack sequence from the 
client before sending any new messages.  Our client (telnet on a PC 
running Windows 2000) is ack'ing packets on a 200 ms latency, so we are 
seeing packet messages at about a 5 Hz rate instead of the 1000 Hz rate 
we were expecting.  No packets are dropped or timed out in the 
transaction sequence, just stopping until an ack is received.

We're wondering if there is something like a water mark or some other 
configuration in the lwipopts.h that might present such a behaviour.   
I've included our settings after the pseudo code below.

Any insight would be greatly appreciated.

Thanks,

-Mike Williamson


-------------------------------------------
Test Pseudo Code

int cnt = 0;
s = socket();
bind(s,&sa,sizeof(sa));
listen(s,1);
c = accept(s,...)
setsockopt(c,... TCP_NODELAY...)
while (1)
{
   char buffer[100];
   sprintf(buffer,"PACKET %d", cnt++);
   send(s,buffer,strlen(buffer),0);
   sleep(1 millisecond);
}


----------------------
Lwipopts.h

/** SYS_LIGHTWEIGHT_PROT
* define SYS_LIGHTWEIGHT_PROT in lwipopts.h if you want inter-task 
protection
* for certain critical regions during buffer allocation, deallocation 
and memory
* allocation and deallocation.
*/
#define  SYS_LIGHTWEIGHT_PROT    1

//----------------------------------------------------------------------
/* ---------- Memory options ---------- */
//----------------------------------------------------------------------

/* MEM_ALIGNMENT: should be set to the alignment of the CPU for which
  lwIP is compiled. 4 byte alignment -> define MEM_ALIGNMENT to 4, 2
  byte alignment -> define MEM_ALIGNMENT to 2. */
#define MEM_ALIGNMENT           4
/* MEM_SIZE: the size of the heap memory. If the application will send
a lot of data that needs to be copied, this should be set high. */
#define MEM_SIZE                3200//1600
/* MEMP_NUM_PBUF: the number of memp struct pbufs. If the application
  sends a lot of data out of ROM (or other static memory), this
  should be set high. */
#define MEMP_NUM_PBUF           16
/* Number of raw connection PCBs */
#define MEMP_NUM_RAW_PCB        4 
/* MEMP_NUM_UDP_PCB: the number of UDP protocol control blocks. One per 
active UDP "connection". */
#define MEMP_NUM_UDP_PCB        6
/* MEMP_NUM_TCP_PCB: the number of simulatenously active TCP connections. */
#define MEMP_NUM_TCP_PCB        16
/* MEMP_NUM_TCP_PCB_LISTEN: the number of listening TCP connections. */
#define MEMP_NUM_TCP_PCB_LISTEN 8
/* MEMP_NUM_TCP_SEG: the number of simultaneously queued TCP segments. */
#define MEMP_NUM_TCP_SEG        16
/* MEMP_NUM_SYS_TIMEOUT: the number of simulateously active timeouts. */
#define MEMP_NUM_SYS_TIMEOUT    16


//----------------------------------------------------------------------
/* The following four are used only with the sequential API and can be
  set to 0 if the application only will use the raw API. */
//----------------------------------------------------------------------

/* MEMP_NUM_NETBUF: the number of struct netbufs. */
#define MEMP_NUM_NETBUF         16
/* MEMP_NUM_NETCONN: the number of struct netconns. */
#define MEMP_NUM_NETCONN        16
/* MEMP_NUM_APIMSG: the number of struct api_msg, used for
  communication between the TCP/IP stack and the sequential
  programs. */
#define MEMP_NUM_API_MSG        32
/* MEMP_NUM_TCPIPMSG: the number of struct tcpip_msg, which is used
  for sequential API communication and incoming packets. Used in
  src/api/tcpip.c. */
#define MEMP_NUM_TCPIP_MSG      32

//----------------------------------------------------------------------
/* ---------- Pbuf options ---------- */
//----------------------------------------------------------------------

/* PBUF_POOL_SIZE: the number of buffers in the pbuf pool. */
#define PBUF_POOL_SIZE          128
/* PBUF_POOL_BUFSIZE: the size of each pbuf in the pbuf pool. */
#define PBUF_POOL_BUFSIZE       2048


//----------------------------------------------------------------------
/* ---------- ARP options ---------- */
//----------------------------------------------------------------------

/** Number of active hardware address, IP address pairs cached */
#define ARP_TABLE_SIZE          20
/**
* If enabled, outgoing packets are queued during hardware address
* resolution.
*
* This feature has not stabilized yet. Single-packet queueing is
* believed to be stable, multi-packet queueing is believed to
* clash with the TCP segment queueing.
*
* As multi-packet-queueing is currently disabled, enabling this
* _should_ work, but we need your testing feedback on lwip-users.
*
*/
#define ARP_QUEUEING                    0
// if TCP was used, must disable this in v1.1.0


//----------------------------------------------------------------------
/* ---------- IP options ---------- */
//----------------------------------------------------------------------

/* Define IP_FORWARD to 1 if you wish to have the ability to forward
  IP packets across network interfaces. If you are going to run lwIP
  on a device with only one network interface, define this to 0. */
#define IP_FORWARD              0
/* If defined to 1, IP options are allowed (but not parsed). If
  defined to 0, all packets with IP options are dropped. */
#define IP_OPTIONS              0
/** IP reassembly and segmentation. Even if they both deal with IP
*  fragments, note that these are orthogonal, one dealing with incoming
*  packets, the other with outgoing packets
*/
/** Reassemble incoming fragmented IP packets */
#define IP_REASSEMBLY                   1 
/** Fragment outgoing IP packets if their size exceeds MTU */
#define IP_FRAG                         1 


//----------------------------------------------------------------------
/* ---------- ICMP options ---------- */
//----------------------------------------------------------------------
#define ICMP_TTL                255


//----------------------------------------------------------------------
/* ---------- RAW options ---------- */
//----------------------------------------------------------------------
#define LWIP_RAW                        1
#define RAW_TTL                        255


//----------------------------------------------------------------------
/* ---------- DHCP options ---------- */
//----------------------------------------------------------------------

/* Define LWIP_DHCP to 1 if you want DHCP configuration of
  interfaces. DHCP is not implemented in lwIP 0.5.1, however, so
  turning this on does currently not work. */
#define LWIP_DHCP               0
/* 1 if you want to do an ARP check on the offered address
  (recommended). */
#define DHCP_DOES_ARP_CHECK     1


//----------------------------------------------------------------------
/* ---------- UDP options ---------- */
//----------------------------------------------------------------------
#define LWIP_UDP                1
#define UDP_TTL                 255


//----------------------------------------------------------------------
/* ---------- TCP options ---------- */
//----------------------------------------------------------------------
#define LWIP_TCP                1
#define TCP_TTL                 255
/* TCP receive window. */
#define TCP_WND                 2048//1024//2048
/* Maximum number of retransmissions of data segments. */
#define TCP_MAXRTX              12
/* Maximum number of retransmissions of SYN segments. */
#define TCP_SYNMAXRTX           4//6
/* Controls if TCP should queue segments that arrive out of
  order. Define to 0 if your device is low on memory. */
#define TCP_QUEUE_OOSEQ         1
/* TCP Maximum segment size. */
#define TCP_MSS                 1024//128
/* TCP sender buffer space (bytes). */
#define TCP_SND_BUF             2048//256
/* TCP sender buffer space (pbufs). This must be at least = 2 *
  TCP_SND_BUF/TCP_MSS for things to work. */
#define TCP_SND_QUEUELEN        4 * TCP_SND_BUF/TCP_MSS
/* TCP writable space (bytes). This must be less than or equal
  to TCP_SND_BUF. It is the amount of space which must be
  available in the tcp snd_buf for select to return writable */
#define TCP_SNDLOWAT            TCP_SND_BUF/2


//----------------------------------------------------------------------
/* ---------- Other options ---------- */
//----------------------------------------------------------------------

/* Support loop interface (127.0.0.1) */
#define LWIP_HAVE_LOOPIF               0

// only one of these two should be set to 1 !!!
#define LWIP_EVENT_API                 0
#define LWIP_CALLBACK_API              1

#if (LWIP_EVENT_API && LWIP_CALLBACK_API)
#error "Can't define both LWIP_EVENT_API & LWIP_CALLBACK_API as 1"
#endif

#define LWIP_COMPAT_SOCKETS             1

// for uC/OS-II port on TI DSP
#define TCPIP_THREAD_PRIO               5
//#define SLIPIF_THREAD_PRIO              1
//#define PPP_THREAD_PRIO                 1
//#define DEFAULT_THREAD_PRIO             1


//----------------------------------------------------------------------
/* ---------- Socket Options ---------- */
//----------------------------------------------------------------------

/* Enable SO_REUSEADDR and SO_REUSEPORT options */
#define SO_REUSE 0