Re: [Bug-apl] Importing large arrays into GNU APL runtime

[Nick Lobachevsky]

I would try to import the matrix in pieces.  Try something like

      mat←951192 10⍴ '' '' '' '' '' '' '' '' 0 0

then import the columns separately, i.e.

      mat[;1]←(↓col1)~¨' '
      mat[;2]←(↓col2)~¨' '

where col1, col2, and friends are simple character matrices containing
data for those columns.

For the numeric columns,you could try something like

      mat[;10]←⍎,' ',col10

Be sure col10 has high minus signs for negative numbers.

Needless to say, everything will have to be the right length, or else
length errors will result at assignment.  This is not a bad way to
check for errors.

Now it is possible that there is a limit to how much execute can
handle, one way around that would be to execute each cell.

      mat[;10]←∊⍎¨↓col10
      mat[;10]←⊃,/⍎¨↓col10 ⍝ if you prefer

An alternative to the execute function is []FI, but I am not sure
whether GNU APL has this.  []FI did character vector to numeric vector
conversion, and a companion []VI function returned a boolean depending
on whether a number was well formed.

Note the situations where you have errors.

I would try to understand what the limits of the system are, namely
the longest string (or more probably the maximum number of tokens)
which can be executed, and also possibly the maximum length of a
function.

On my older Dyalog APL system, adding one resulted in a LIMIT ERROR

      ⍴⍎⍕⍳315465

On 2/11/14, Elias Mårtenson <address@hidden> wrote:
> I wanted to use GNU APL to work with a fairly large dataset. The data is
> statistical output from a benchmark and it's 951192 rows and 10 columns. 8
> of the columns are text (around 10 or so characters each) and the remaining
> two are numeric. The data is in a single text file.
>
> I have had no end of trouble getting this data into APL. I tried generating
> a function and execute it, but that caused GNU APL to crash.
>
> I'm at a point where I am considering writing a function using libfile_io
> to read the data, but I'm curious as to what the "proper" solution is
> supposed to be.
>
> List members, what is your take on this?
>
> Regards,
> Elias
>