Re: GigE (was Re: [Discuss-gnuradio] DSP based SDR)

[Harald Welte]

On Tue, Jun 21, 2005 at 04:10:35PM -0700, John Gilmore wrote:
> I'm picking up the Gigabit Ethernet conversation from a few weeks ago.

great :)


> > I'm by no means arguing that the linux network stack is slow.  It's just
> > not intended for an application which just wants to get big data streams
> > with low latency and little overhead into userspace.
> 
> I'm surprised to hear you suggest this (though I note you didn't give
> numbers, just generalizations like "enormous").  

I didn't state (or at leat want to state) that it was impossible to
achieve multiple-gigabit data rates with the Linux network (or even
TCP/IP) stack.

I just think that it is a horrible waste of cpu cycles and memory
bandwidth, when all you want is to get large amounts of data from a
local (cloe to the computer) device into an userspace application.  This
is especially true if your userspace application is a cpu hog like
digital signal processing algorithms ;)

> Ethernet.  The last time I looked at the Linux network stack was when
> Dave Miller was cutting his teeth on the SPARC Linux port and making
> 100MB/s Ethernet run within a few % of theoretical maximum (he quit
> when he exceeded 10MBytes/sec over TCP from user process to user
> process).  I had of course expected that somebody would do the same
> thing for Gigabit Ethernet.  This appears to have been done; GigE
> cards seem to be able to push large amounts of data under Linux.

sure. no problem with that.  

> What thruput do YOU see using a good GigE card under Linux?  Can you move
> 100 Mbytes/sec from user process to user process with TCP, across two GigE
> interfaces and a GigE switch?  If so, it's doing much better than USB2
> ever will, and much better than any version of FireWire.

I'm usually not doing client-to-server TCP benchmarks over my test
networks, since I'm more interested in small-packet forwarding
performance.  I'm sure with the right tuning (and probably some of the
recent work on TCP BIC etc.) you can saturate a 1000-base-tx link with a
single tcp flow, no question on that. 

But at what expense?  you copy data back and forth between different
address spaces (kernel/user), you have lots of complex code that deals
with retransmissions, protocol demultiplex, ... that is totally
unneccessarry for the gnuradio/USRP kind of application.
 
> This msg says that with a 10GB/s Ethernet card (note: 10x as fast as GigE)
> they can send UDP at 1.9GBps, and send TCP at 2.2Gbps from Chicago to
> Switzerland.  Clearly the Linux kernel overhead isn't keeping thruput
> below a gigabit:
>   http://www-iepm.slac.stanford.edu/monitoring/bulk/10ge/

No.  But in those tests, transferring data is the _real_ application to
which you want to dedicate all resources.  For USRP, the data transfer
is merely a small step, a 'necessarry prerequisite'.

> There is a lot of research about how to make TCP run at gigabit speeds
> over *long distances*, particularly in recovering from dropped packets.

yes, I'm very well of what's going on in this area (especially since the
first results of this work are already merged in recent 2.6.x mainline
kernels).

> Harald, in what way is your experience different?

Hope my email clarifies :)

-- 
- Harald Welte <address@hidden>                         http://gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)

signature.asc
Description: Digital signature