Re: reading data from ascii files.

[Jaroslav Hajek]

On Fri, Jul 16, 2010 at 5:32 PM, Manoj Deshpande
<address@hidden> wrote:
> I have a very similar problem, slightly more complicated i would think. I
> have 7 text files, which i read sequentially, each file has about 400,000
> rows, i had written script which had nested loops , and it took over 3 days
> for my script to complete. I quickly changed the scope of my script, and
> also decided avoid nested loops , since i read it some where on this forum,
> that nested loops can be a problem. yet i seem to get very slow processing
> for these 3million rows.
>
> End motive : loop over  7 text files, in each file, read one row at a
> time(while(fgets)and dlmread), identify if row is useful, append one of the
> column values into one of the 12 matrices i have, at the end of reading all
> rows, draw histograms, one for  each matrix,
>
> Problem : No prior knowledge of size of matrices, or the number of rows in
> all the files.
> And yes i am doing all this on RH7 linux.
>
> Is there a way to quicken the read and append to matrix, whose sizes cant be
> pre-determined ?
>
> Thanks,
> Manoj
>

Note that appending to a vector usually requires reallocating storage
and copying the whole vector to the new storage.
If you append a single element to a vector repeatedly using
v(end+1) = x;
then Octave 3.2+ will detect this as a stack usage and attempt to
reduce the copying by over-allocating storage in advance, much like
std::vector in C++ does. This doesn't work for rows or columns,
however, so in that case the best approach is to collect the rows into
a cell array and concatenate them afterwards.

Show some code if you want more detailed advice.

-- 
RNDr. Jaroslav Hajek, PhD
computing expert & GNU Octave developer
Aeronautical Research and Test Institute (VZLU)
Prague, Czech Republic
url: www.highegg.matfyz.cz