Re: [Discuss-gnuradio] Reliable Ethernet controllers and laptops for USRP N210 ?

[Juha Vierinen]

Hi,

I have also had a long and continuing fight with NICs and dropped
packets. Typically all Intel PCIExpress cards have been good, but
recently I have started having problems with Intel too. For example,
Intel 82579V consistently drops packets. I haven't seen any PCI bus
card work at full 25 MHz bandwidth either. The problem doesn't seem to
be USRP specific either. I also have problems with dropped packets on
other high bandwidth UDP transport applications (e.g., on my LOFAR
4*1Gbit data collection software). The rate of dropped packets is
typically not very high (maybe one out of 10 million), but it is still
annoying as I have many systems that don't drop any packets over the
course of days of continuous full bandwidth operation.

I have tried many things, such as forcing CPU in performance mode,
increasing recv_buff_size, recv_frame_size, and using the latest intel
drivers. None of these solve the problem completely. In all of the
cases, my benchmark is a nullsink type of an application that just
reads incoming packets and does nothing with the data. The CPU usage
has also been less than 100% per core in all cases.

However, I have recently found a possible fix. I stumbled upon some
intel documentation on ethernet card performance about two weeks ago.
It had a section about the InterruptThrottleRate feature of the intel
driver. The intention of this is to reduce the number of interrupts
per second to ease the load on the poor CPU trying to handle all the
incoming packets. By default, the feature is set to something called
"conservative". This raised my suspicions, so I tried disabling it.
This supposedly has the effect of reducing latency.  On my linux +
intel quad gigabit ethernet card setup the relevant commands are:

> rmmod e1000e
> modprobe e1000e InterruptThrottleRate=0,0,0,0

After this my LOFAR data collection server runs on a single six core
PC and quad gigabit ethernet system with full 400 MB/s bandwidth
without any dropped packets (I have yet to encounter one, even after
days of operation). This is using an Intel 82571EB. I didn't have a
noticable increase in CPU load either. I have also tried the same
option with USRPs and the problem also seems to go away. I haven't
done a conclusive testing yet with USRPs, so I cannot say for sure
that this helps in all cases. I also haven't had a chance to test this
on a 82579V either.

juha

On Wed, Dec 12, 2012 at 9:00 PM, Rickard Radio <address@hidden> wrote:
> OK, I see.  However strange is that I can get UDP traffic to work up to
> about 925 Mbit/s without any dropped packets when testing with "iperf" and
> Ethernet cable directly between the two Acer laptops. That with tuning UDP
> buffer size to 600 kB and datagram packet size to 3000 bytes (using "-l" and
> "-w" iperf parameter options). But trying to use the same tuned values, with
> UHD (I assume corresponding) parameters "recv_buff_size"  and
> "recv_frame_size" does not work. But maybe I misunderstand or mix up
> something, chasing lost packets in the dark.
>
> On Dec 12, 2012, at 7:00 PM, address@hidden wrote:
>
> Rickard:
>
>
>
> By "silently ignore", I mean that they'll apparently take the higher MTU
> setting, but will actually drop any frames that exceed the default (1500)
> MTU.  Which is a particularly insidious failure mode for these drivers...
>
>
>
>
> On 12 Dec 2012 12:46, Rickard Radio wrote:
>
> OK, thanks. My problematic Acer with AR8151 NIC accepts MTU's up to about
> 6100. Trying to set higher MTU it responds with an error. After setting to a
> large value, such as 6000, I checked with ifconfig whether it was accepted.
> However, as said, it has no effect on the failing UDP communication with the
> N210.
>
> Rickard
>
> On Dec 12, 2012, at 6:38 PM, address@hidden wrote:
>
> MANY NICs don't support larger frame sizes.  Some of them require that you
> set the larger MTU using ifconfig, and some will error-out if you ask for an
> MTU too large.  Others will accept the larger MTU request and then silently
> ignore it.
>
>
>
> On 12 Dec 2012 12:22, Rickard Radio wrote:
>
> Thanks!
>
> This is really valuable information. Also your new document with lots of
> useful info and explanations what happens behind the scenes.
>
> However, on my problematic Acer Aspire 8820T laptop with AR8151 NIC and
> "atl1c" driver (instead of the Intel "e100e" driver below) , nothing of this
> has an effect. For example, sudoing has no effect as well no effect of
> changing the frame sizes, CPU governor, receive buffer size, etc. Played
> with most of this before (see thread http://www.ruby-forum.com/topic/3949213
> )
>
>
> sudo /usr/local/share/uhd/examples/benchmark_rate --rx_rate 25e6 --tx_rate
> 25e6 --duration 30 --args="recv_frame_size=3792,send_frame_size=3792"
>
>
> For this to have an effect of frame size, I need also to specify the
> ip-address (otherwise the arguments seem to be ignored):
> sudo /usr/local/share/uhd/examples/benchmark_rate --rx_rate 25e6 --tx_rate
> 25e6 --duration 30 --args="addr=192.168.10.2,
> recv_frame_size=3792,send_frame_size=3792"
>
> In any case, connection stalls/hangs with repeated "Error code 1: Unexpected
> error on recv, continuing…" messages with the "benchmark_rate" test.
> Similar thing if I run uhf_fft at the same samp rate, it just stalls, orange
> LED starts rapidly blinking and the "C" LED goes off, and the FFT plot just
> freeze since no more samples are coming from the N210.
>
> So I suppose something is terrible wrong with the AR8151 NIC, or internally
> further down the pipe at high sampling rates. I have now finally given up on
> those Acer machines for these high samp rates.
> Now writing to "SDR Santa" to bring me more powerful Lenovo Thinkpad T430s
> needed for my high sample rate experiments and projects.
>
> Rickard
>
>
> On Dec 11, 2012, at 9:53 PM, Balint Seeber <address@hidden> wrote:
>
> Hi Richard,
>
> Thanks for reporting on your experiments! I have doing a few lately myself
> (specifically in regards to USRP latency).
>
> I have a Thinkpad T430s with a 82579LM. The benchmark_rate reports zero
> underflows/overflows for unidirectional streaming. Bi-directional results in
> some TX underflows with the default MTU, however increasing this results in
> no underflows, e.g.
>
> sudo ifconfig eth0 mtu 9000
>
> sudo /usr/local/share/uhd/examples/benchmark_rate --rx_rate 25e6 --tx_rate
> 25e6 --duration 30 --args="recv_frame_size=3792,send_frame_size=3792"
>
> Remember the 'sudo' will help by enabling 'real-time' (SCHED_FIFO)
> scheduling in the Linux kernel.
>
> Benchmark rate summary:
>   Num received samples:    749991475
>   Num dropped samples:     0
>   Num overflows detected:  0
>   Num transmitted samples: 749969786
>   Num sequence errors:     0
>   Num underflows detected: 0
>
> I have found a number of parameters can be tweaked to change general
> N-series communication behaviour with the Intel NIC (specifically interrupt
> moderation, MTU [above], etc):
>
> sudo modprobe -r e1000e
> sudo modprobe e1000e debug=16 InterruptThrottleRate=0 TxAbsIntDelay=0
> TxIntDelay=0 RxAbsIntDelay=0 RxIntDelay=0
>
> Changing CPU governor to 'performance' on all cores may also help in certain
> situations.
>
> Please take a look at this document I've been putting together for more
> details: http://code.ettus.com/redmine/ettus/projects/public/wiki/Latency
>
> Let us know if any of the above tips improve your results.
>
> Hope that helps,
> Balint
>
> On Tue, Dec 11, 2012 at 8:26 AM, Rickard Radio <address@hidden>
> wrote:
>>
>> OK, I just tried this on a colleagues brand new Thinkpad T430s and it
>> works without any packet drops or other Ethernet problems on that laptop (at
>> least for a minute or so). It has the Intel 82579LM Ethernet chip so that
>> chip doesn't seem too bad after all (like the Artheros 8151 which I have
>> problems with).
>>
>> Rickard
>>
>> On Dec 11, 2012, at 12:51 PM, Rickard Radio <address@hidden>
>> wrote:
>>
>> Hi all,
>>
>> Can anyone please try this on a laptop, preferably a Lenovo Thinkpad, with
>> a N210 and report what you get:
>> "/uhd/examples/benchmark_rate --rx_rate 25e6"
>>
>>
>> Thanks
>> Rickard
>>
>> On Dec 7, 2012, at 3:32 PM, Rickard <address@hidden> wrote:
>>
>> I wonder which GigE controllers (manufacturer & versions) and laptop
>> computers are working hassle free, without dropping packets etc., at the
>> highest sampling rates with the USRP N210 which results in 800 Mbit/s raw
>> data (or about 835 Mbit/s total UDP) ?!
>>
>> Please test your laptop with  "/uhd/examples/benchmark_rate --rx_rate
>> 25e6" or "uhd_fft -s 25e6"  and report your findings!
>>
>>
>> _______________________________________________
>> Discuss-gnuradio mailing list
>> address@hidden
>> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>>
>
>
> _______________________________________________
> Discuss-gnuradio mailing list
> address@hidden
> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>
>
>
>
>
>
> _______________________________________________
> Discuss-gnuradio mailing list
> address@hidden
> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>