Re: Is there something quicker than textscan?

[Nicholas Jankowski]

just curious, for block datafiles I often just use some variant of
load -ascii.  how does this compare with dlmread, other than I can
specify a separator and I guess just part of the file? for speed I
noticed on older versions that the various formatted text read
commands were a bit slower than just loading the whole file and
assigning columns to different variables after the load.  those were
smaller files, though, and now the larger files I work with (maybe
1000x14 or so) I just do it all with load.

nickj

On Mon, Mar 19, 2012 at 8:35 AM, Reza Housseini <address@hidden> wrote:
>
>
> On Mon, Mar 19, 2012 at 12:44 PM, PhilipNienhuis <address@hidden>
> wrote:
>>
>>
>> maiky76 wrote
>> >
>> > Hi,
>> >
>> > I have to load quite a few files from an analyzer (.txt with 4 columns
>> > of
>> > 16384 lines of fft: freq, Gain, Phase and Coherence with plus a footer
>> > with the analyzer settings that i don't care about).
>> >
>> > I use the following code in a loop to store the data before performing
>> > some calculation in the loop:
>> >
>> > [...]
>> >
>> > fid   = fopen(D_FR(k).name);
>> > data  = textscan(fid,'%f %f %f %f',16384);
>> > fclose(fid);
>> >
>> > FRTF_gain(:,k)        = data{1,2};
>> > FRTF_phase(:,k)  = data{1,3};
>> > FRTF_coh(:,k)     = data{1,4};
>> >
>> > [...]
>> >
>> > The slowest step by (very) far is the textscan process.
>> > I average 85s per file with Octave 3.6.1 and 0.25s with Matlab 2010a.
>> >
>>
>> Yeah, textread and textscan have a pathetic performance when a format
>> repeat
>> count is specified on relatively long files. The cause lies in about the
>> only piece of original code I haven't touched yet.
>>
>> I have a patch in mind (and tried in a debug situation) that will
>> significantly speed up your problem (though not as fast as ML), but that
>> will have to wait until (1) I got time to implement it and (2) release of
>> Octave 3.8. Or maybe 3.6.2 if it the core developers agree that the poor
>> performance is a bug (I'd say so but who am I).
>>
>> This situation will only get better as soon as a binary textscan has been
>> implemented. It is in the making but it will probably take some more time.
>>
>>
>>
>> > Question: how could I load the data quicker than that in Octave?
>> >
>>
>> As I answered a few days go in a thread with a similar subject, Octave
>> offers many text I/O solutions, not just textscan.
>> In your case:
>>
>> C = dlmread (D_FR(k).name, ' ', 'A1:D16384')
>>
>> ...probably works fine.
>>
>> (see 'help dlmread')
>>
>> In a general sense, don't complain too soon that "ML does it
>> better/faster/....". That may be true for that specific literal piece of
>> ML
>> code, but depending on the situation Octave offers alternative solutions
>> may
>> be on par or sometimes even better.
>>
>> Philip
>>
>>
>> --
>> View this message in context:
>> http://octave.1599824.n4.nabble.com/Is-there-something-quicker-than-textscan-tp4483991p4484467.html
>> Sent from the Octave - General mailing list archive at Nabble.com.
>> _______________________________________________
>> Help-octave mailing list
>> address@hidden
>> https://mailman.cae.wisc.edu/listinfo/help-octave
>
>
> I used dlmread for files with more then 35000 lines. There's no remarkable
> delay and it fits perfectly for your purpose.
>
> _______________________________________________
> Help-octave mailing list
> address@hidden
> https://mailman.cae.wisc.edu/listinfo/help-octave
>