Re: [Discuss-gnuradio] disk throughput

[Eric Blossom]

On Thu, Dec 30, 2004 at 05:13:11PM -0500, cswiger wrote:
> On Thu, 30 Dec 2004, Dan Gisselquist wrote:
> 
> > Looking closely at this problem, it looks like you may have buffer
> > problems--that is, you may be missing buffers by not keeping up with
> > the raw data rate.
> >
> 
> Uh yep - as much as I would like to blame my problems on
> the software ;)  -  facts kept pointing to my ADC data, and
> looking closer turns up discontinuities about every 8960/8972
> samples. Damn. Back to basic signal acquisition.
> 
> --Chuck

I'm not sure of your data acquistion rate, but when I was working on
the ATSC receiver we used to stream data to the disks at 40MB/sec for
periods of an hour or so at a time with no overruns.

It took a bit of doing.  Here are some of the things we learned:

(1) The ext3 file system is the wrong answer.  It posts it journal
    every now and then (something like 5 seconds), and while it's
    doing that you're screwed.  Mounting the ext3 file system as ext2
    worked.  No journaling, more consistent throughput.  You may want
    to try the xfs file system too.

(2) We used two Western Digital drives (they were 120GB, now I'd
    probably go for the 200 or 250 GB drives).  We used the Linux
    Logical Volume Manager (LVM) to setup a software RAID 0
    configuration.  We striped the two drives (actually two
    partitions, one on each drive) with a large block size.  It was at
    least 64K, could have been more.

(3) Used a 64-bit 66-MHz IDE controller.  The current generation of 
    on-the-motherboard ones may be fast enough.  We used one of the
    Promise Technology boards, an Ultra133TX2.

I also did some experiments using 36 GB IBM Ultrastar 10K RPM SCSI
drives.  Striping three of them and using a 64-bit 66-MHz dual channel
Ultra-160 SCSI controller I could sustain 160 MB/sec till I filled the
disks.  This configuration didn't get along nicely with the A/D board,
so I ended up using the IDE drives instead.  Adding a fourth drive
didn't help throughput much.  The bottleneck at that point was
somewhere else in the system.

Eric