Re: File size limits in textread()?

[Stefan van der Walt]

Here's a much faster version of the code posted earlier.

I tested this on 60000 lines of text:

octave:3> a = cputime(); [x,y,z,p] = tread('large.txt', '%s %s %d %d'); 
cputime() - a, size(x), size(y), size(z)
ans = 0.88287
ans =

      1  59999

ans =

      1  59999

ans =

      1  59999


Regards
Stefan

On Fri, Jun 10, 2005 at 12:53:57PM +1000, address@hidden wrote:
> G'day all,
> 
> I'm using textread() to read in mixed string and integer data from a space 
> delimited text file. I found it worked okay with one data file, but then 
> hangs when reading another,  larger data file. Hitting ctrl-c brings back the 
> octave prompt.
> 
> Any ideas as to why textread() doesn't like the larger data file? This file 
> has about 6000 rows and 10 columns, 4 of which are strings, the others are 
> integers.
> 
> I'm now using a loop with fgetl() and split(), but it's very slow.
> 
> The data file is generated from a database query. I might use 2 queries, one 
> for integer data and one for string data. That way at least I can use load or 
> dlmread() to quickly pull in the integer data, and then deal with the string 
> data using textread().
> 
> -------
> 
> BTW, the data file is generated from a database query. I had earlier asked 
> about interfacing octave to databases (postgreSQL) as I had problems with 
> some interface code from Dirk Eddelbeuttel. I believe that was because the 
> postgresql version I'm using, 7.2.1, was built using g++ 2.95, i.e. shared 
> libraries for c++ interfacing had unresolved symbols for this reason--lack of 
> agreed ABI. I'll try again at some stage with 7.4, but I have a 5GB database 
> that will have to be converted before I can do that...takes ages.
> 
> Cheers,
> 
> Paul Koufalas
> Adelaide, S. Australia
>

tread.cc
Description: Text document