Re: [lwip-devel] Packet queues (was Re: [bug #11400]ARP multi-packet-queue modifies TCP ...)

[Jonathan Larmour]

Goldschmidt Simon wrote:
> Hi,
>
> sorry I was posting to the bug yesterday evening, I just did not see 
> these mails...
>
>  >The issue being that most sensible drivers will not poll and wait for the
>  >ethernet driver transmit to complete, but instead will hand off the packet
>  >data to the driver. Some of my hardware drivers copy the data first due to
>  >hardware restrictions, but that's time-consuming and more importantly 
> takes
>  >more memory - one MTU size buffer for each tx buffer I allow outstanding.
>  >For the others I just point the driver at the pbuf chain, after having
>  >given it a pbuf_ref() first; but now I realise that is insufficient since
>  >control may be returned to the application before the packet has truly 
> been sent.
>
> I solved this in 2 ways:
> 1. I call pbuf_ref() on each pbuf I'm sending, calling pbuf_free() when 
> sending is finished. For that to work, a TX-interrupt is needed and I 
> need to work out the pbuf pointer from the data pointer (can be tricky).
> That's why I was pushing on not to change a pbuf's contents while ref > 1.

Indeed, hence me raising this issue - the API doesn't prevent users 
changing pbufs even if the ref > 1. There's nothing about users checking 
the reference count.

> 2. With a PBUF_REF, I simply 'hope' the ethernet hardware is fast enough 
> (I have only 20 packets on queue max, that's about 2.5 ms to send all 
> packets)
> This is true for _my_ system (which is a slow Altera NIOS-II @ only 60 
> MHz), also because the queue is empty most of the time which means the 
> packet is sent with no delay. But it can (and will) be different on 
> faster systems or 10-mbit ethernet.

It can be true on any RTOS where network operations can be pre-empted by 
higher priority tasks - perhaps even the thread that is sending the data 
will be a high priority task preventing execution of the lwIP tcpip thread.

> Generally, does it mean we have to copy each PBUF_REF (and PBUF_RAM 
> coming from the application?) using memcpy() in the ethernet driver?
> If so, it may be faster to wait in the network driver until the packet 
> is finished sending... Not that I favour it, but copying using the 
> processor is a lot slower in my system...

The purpose of my suggested addition is to avoid the copying, if the users 
indicate it can be avoided.

>  >>True, but there are very few standard APIs that are capable of it, which
>  >>is probably why there hasn't been much concern about it.
>
>  >But I had thought it was meant to be a feature of the lwip API :).
>
> And if others can do it, why can't we? :)

Exactly!

>  >>Zero-copy
>  >>through the sockets API is nigh on impossible for example.
>
> On RX side, it's impossible, but on TX side, why not? As long as the 
> data is not being queued... (that takes out TCP, of course!)

On RX it's definitely possible! It does depend on hardware capabilities 
obviously, but it's easy on many ethernet MACs to DMA directly into the 
next PBUF_POOL pbuf (which you would have to have preallocated). Sometimes 
you may have to ensure the pbuf pool is correctly aligned. Sometimes the 
PBUF_POOL_BUFSIZE will need to be large enough for a whole frame (e.g.1518 
bytes), although some hardware can do scatter-gather with smaller chunks as 
needed. But it's definitely possible.

Even for TCP it can offer performance and memory improvements - your ARP 
multi-packet queue implementation will have to copy pbufs, because it 
doesn't know where the pbufs came from, and in the case of TCP, copying is 
pointless because the data _won't_ change because it's in a queue higher up.

Jifl
--
eCosCentric Limited      http://www.eCosCentric.com/     The eCos experts
Barnwell House, Barnwell Drive, Cambridge, UK.       Tel: +44 1223 245571
Registered in England and Wales: Reg No 4422071.
  **  Visit us at ESC Silicon Valley <http://www.embedded.com/esc/sv> **
  **  April 3-5 2007, Booth 1922, San Jose McEnery Convention Center  **
------["The best things in life aren't things."]------      Opinions==mine