help-octave
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Importing large amounts of data


From: Hannes
Subject: Re: Importing large amounts of data
Date: Tue, 19 Jun 2012 04:53:55 +0200
User-agent: Dynamic Internet Messaging Program (DIMP) H3 (1.1.4)


Quoting Hannes <address@hidden>:

Quoting "Francesco Potortì" <address@hidden>:

Right now I am trying to do this with a 150x150x1000 int array. This
array has a small memory footprint in C++ and the file being pushed from
the C++ program to the octave script is around 65MB.

Those are 22.5e6 elements.  If you are using a binary representation
with 4-byte integers, you should have a 90-MB file.  If you use 16-bit
integers, half that measure.  65 MB, if there are no errors, indicates
that you are using a text representation, which is good and easy to
debug for your case, but may become slow if you are planning to use much
bigger arrays.
                                                      When reading this
into Octave it already consumes 8GB of RAM, which is quite a surprise,

Octave uses 8-byte floats by default, but it can read and write 1, 2, 4
and 8-byte integers.  Even when using the default, your array should
consume around 180 MB.  If you see 8 GB, something is going wrong.

but not the main problem (I have memory to spare right now). However the
reshaping is already going of for two days now on a multi-cpu Xeon Server.

This too is strange.  Should be in the order of a few seconds at most.

Whats going wrong? How should I approach this to get it done?

Tell us exactly what format you are using for writing the file (an
example with a small array will suffice) and what commands exactly you
use for reading it in.  For example, try with a 2x2x3 array first.

Well, like I said, I am writing this as code, that the Octave script imports using source().

I have attached a compressed file that is 2.5MB unpacked. The big array in this file has dimensions 150x150x55 (roughly). This file takes about 20s to import on my Workstation.
If I export a larger array, like I said, it gets really slow.


[now with attachment, sorry for double-posting]

Attachment: example.m.xz
Description: application/xz


reply via email to

[Prev in Thread] Current Thread [Next in Thread]