Reading ASCII data of unknown length (summary)

[Tim Kerwin]

Thanks to all who responded to my query about reading ASCII files of unknown 
length into octave.  Most of the solutions offered involved invoking "wc" on 
the file (via system()) in order to get the number of lines in the file.  Some 
proceed with something like:

   for i=1:nlines
      [x(i) ... ] = fscanf(...

but this is very slow.  Better appears to be to tack an octave header on to 
the ASCII file, then use "load", which is much faster than repeated fscanf.  
Here's the function "aload" (adapted from one sent by Eyal Doron 
<address@hidden> -- I just changed the temporary file name to use 
a process id number so users don't stomp on each other's files):

function [X]=aload(filename,nr)
% function [X]=aload(filename,nr) loads a flat ASCII file into X. If the
% number of rows nr is not specified, will read in the whole file.

if (~exist(filename))
        error("File not found")
end

nc=shell_cmd(["head -n 1 " filename " | wc -w | awk '{printf $1}'"],1);
if nargin<2
  nr=shell_cmd(["wc -l " filename " | awk '{printf $1}'"],1);
end

tmpfile = sprintf("/tmp/oct%s", shell_cmd("echo -n $$"));
fid=fopen(tmpfile,"w");
fprintf(fid,"# name: localX\n# type: matrix\n");
fprintf(fid,["# rows: " nr "\n# columns: " nc "\n"]);
fclose(fid);

if nargin<2
  shell_cmd(["/bin/cat " filename " >> " tmpfile]);
else
  shell_cmd(["/usr/bin/head -n " num2str(nr) " " filename  ...
          " >> " tmpfile]);
end
  
eval (["load -force " tmpfile]);

shell_cmd(["/bin/rm -f " tmpfile]);

X=localX;