[lwip-devel] Packet queues (was Re: [bug #11400] ARP multi-packet-queue modifies TCP ...)

[Jonathan Larmour]

[ Taking this out of the bug to lwip-devel instead .... ]

Kieran Mansley wrote:
> An external application shouldn't be using PBUF_RAM buffers to refer to its
> own memory; it should instead use PBUF_REF (if the memory contents can
> change) or PBUF_ROM (if it can guarantee it will be preserved as long as lwip
> needs it).  PBUF_RAM should only be used to refer to buffers whose payload
> points to a region of memory allocated by lwIP.  However, this might not be
> relevant because on return the API does currently seem to give the
> application the right to do what it likes with the pbuf, so I think we will
> have to copy if we can't send it immediately.  
>
> We could add complexity/efficiency by working out if it's a TCP or UDP
> packet, and then not doing as much copying in the TCP case, but for the first
> pass I think that is unnecessary.  Let's get it working first.
>
> Another alternative would be to extend the API to add the copy/no-copy flag
> for UDP so that we can tell if the application is going to try and reuse the
> buffer when the udp_send() call returns, but again I think we should get it
> working first and worry about making it more efficient later.

I've been thinking about this more generally. How do most people implement 
their ethernet drivers? I've only just realised that at least in my own 
current implementation, there is a brief window when sending data where 
there could be a problem here. I suspect I may be far from alone.

The issue being that most sensible drivers will not poll and wait for the 
ethernet driver transmit to complete, but instead will hand off the packet 
data to the driver. Some of my hardware drivers copy the data first due to 
hardware restrictions, but that's time-consuming and more importantly takes 
more memory - one MTU size buffer for each tx buffer I allow outstanding. 
For the others I just point the driver at the pbuf chain, after having 
given it a pbuf_ref() first; but now I realise that is insufficient since 
control may be returned to the application before the packet has truly been 
sent.

So is zero copy at present impossible to implement safely in lwIP? That's 
bad if so, as one of the good things about zero copy is not just 
performance but memory use. This is very wasteful given there being plenty 
of DMAing controllers out there now capable of zero copy.

If this is the case, then I think we're going to have to improve the API. 
Rather than trying to retain the copyable state through an API parameter 
passed all the way through (like the copy flag to tcp_write), or break the 
API, it may be better to add a new pbuf type PBUF_RAM_NOCOPY.

So we would have:
PBUF_RAM: memory allocated from within lwip. May be changed after return 
from calls. Application frees.
PBUF_REF: memory allocated from application. May be changed after return 
from calls. Application frees.
PBUF_RAM_NOCOPY: memory allocated from within lwip. Application hands over 
the pbuf entirely for the stack to free and does not touch it any more once 
passed in.

Thoughts?

Jifl
--
eCosCentric Limited      http://www.eCosCentric.com/     The eCos experts
Barnwell House, Barnwell Drive, Cambridge, UK.       Tel: +44 1223 245571
Registered in England and Wales: Reg No 4422071.
  **  Visit us at ESC Silicon Valley <http://www.embedded.com/esc/sv> **
  **  April 3-5 2007, Booth 1922, San Jose McEnery Convention Center  **
------["The best things in life aren't things."]------      Opinions==mine