Re: GigE (was Re: [Discuss-gnuradio] DSP based SDR)

[John Gilmore]

I'm picking up the Gigabit Ethernet conversation from a few weeks ago.

Harald Welte said:
> I'm actually not referring to the Ethernet hardware side, but to the
> software stack.  Unless you want to write a special 'ursp network stack'
> that sits directly on top of the hardware driver (like PF_RING, but
> that's read only), you will go through the whole 'normal' Linux network
> stack.
> 
> For every packet you process, there's an enormous amount of overhead
> you're going through, even if you choose to not use IP and play some
> tricks with raw sockets.   Due to the sockets-based interface, you will
> have additional copying of the data.
> 
> I'm by no means arguing that the linux network stack is slow.  It's just
> not intended for an application which just wants to get big data streams
> with low latency and little overhead into userspace.

I'm surprised to hear you suggest this (though I note you didn't give
numbers, just generalizations like "enormous").  The first time I
looked seriously at a network stack was in a Van Jacobson/Mike Karels
class about how they made BSD TCP on Sun-3/50s run at 1 MByte/sec,
within a few % of the theoretical maximum thruput of 10Mbit/s
Ethernet.  The last time I looked at the Linux network stack was when
Dave Miller was cutting his teeth on the SPARC Linux port and making
100MB/s Ethernet run within a few % of theoretical maximum (he quit
when he exceeded 10MBytes/sec over TCP from user process to user
process).  I had of course expected that somebody would do the same
thing for Gigabit Ethernet.  This appears to have been done; GigE
cards seem to be able to push large amounts of data under Linux.

What thruput do YOU see using a good GigE card under Linux?  Can you move
100 Mbytes/sec from user process to user process with TCP, across two GigE
interfaces and a GigE switch?  If so, it's doing much better than USB2
ever will, and much better than any version of FireWire.

Here's a msg that says Linux was getting 100MBps (presumably MBytes/s)
on 2.6.0-test2 in 2003:
  http://www.uwsg.iu.edu/hypermail/linux/net/0308.1/0037.html
Here's one that says 930 Mbps using 2.4.21 on a BiXeon in 2004:
  http://oss.sgi.com/projects/netdev/archive/2004-04/msg00474.html
Here's one that shows 870 Mbits/s in 2002 on 2.4.18 (long haul from
Sunnyvale to Chicago):
  http://www-iepm.slac.stanford.edu/monitoring/bulk/fast/

This msg says that with a 10GB/s Ethernet card (note: 10x as fast as GigE)
they can send UDP at 1.9GBps, and send TCP at 2.2Gbps from Chicago to
Switzerland.  Clearly the Linux kernel overhead isn't keeping thruput
below a gigabit:
  http://www-iepm.slac.stanford.edu/monitoring/bulk/10ge/

There is a lot of research about how to make TCP run at gigabit speeds
over *long distances*, particularly in recovering from dropped packets.
Nobody seems to need to do research on how to make TCP run at gigabit
speeds across the room; it already works.  As does UDP.

Harald, in what way is your experience different?

        John