[lwip-users] raw api tcp_write()ing small chunks fills snd_queue

[Sergio R. Caprile]

Hi, I need help here.
I have this raw api application which tcp_write()s 5 small (~20 bytes)
messages and then checks tcp_sndbuf() to send the biggest possible
chunk, but it ends up having to send a smaller chunk because an internal
queue gets full.

My settings are (almost) standard:
TCP_MSS = 536
TCP_SND_BUF = 2*TCP_MSS
TCP_SND_QUEUELEN = (4*TCP_SND_BUF + TCP_MSS -1)/TCP_MSS
TCP_OVERSIZE = TCP_MSS
MEMP_NUM_PBUF = 30
MEMP_NUM_TCP_SEG = 16
PBUF_POOL_SIZE = 32
PBUF_POOL_BUFSIZE = TCP_MSS+overhead (default)

My operation:
Get a request
tcp_write() with apiflags=0; 5 times, with ~20-byte messages
len = tcp_sndbuf(pcb);
len = LWIP_MIN(len, 2*tcp_mss(pcb));
tcp_write(,len,0)
    900-bytes fails, 450 fails, 225 succeeds

debug log is: "tcp_write: queue too long 9 (8)", that is, line 581 of
tcp_output.c (in 1.4.1 release), Phase 3: Create new segments.

As I understand, those small nocopy tcp_writes generate a queue of type
PBUF_REF, which added to something (? there is no prior traffic in this
pcb, only a just recently closed connection for a prior request)  leads
to a queue that is too big for this segment.

Instead of just increasing TCP_SND_QUEUELEN, or avoiding those small
writes (which I can't right now), I'd like to understand what is going
on and don't have the same problem later.

I've tried inserting a tcp_output() call between the small writes and
the big one, with no difference. I guess this is because tcp can't get
rid of the chain until the remote end acknowledges this segment and I'm
still holding the CPU.
I've set TCP_WRITE_FLAG_COPY for the small writes and it works OK (and
this is what I would do if there is no other option to try)

I'd like to hear suggestions, explanations, criticism, pointers to
documentation I overlooked, whatever. Everything is welcomed.

Regards

--