[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
RE: memory exhausted or requested size too large for range of Octave's i
From: |
William Krekeler |
Subject: |
RE: memory exhausted or requested size too large for range of Octave's index type |
Date: |
Mon, 24 Oct 2011 13:18:55 +0000 |
-----Original Message-----
From: address@hidden [mailto:address@hidden On Behalf Of Ganesh7789
Sent: Saturday, October 22, 2011 4:02 AM
To: address@hidden
Subject: memory exhausted or requested size too large for range of Octave's
index type
I'm new to this. I was trying to read 4.5GiB file which is in csv format. But
my RAM is 3GiB. I'm runnig fedora 15. I have core i5 first generation. I
wanted to convert it to binary format. Can anyone tell me how to do that
using Octave?
Thanks in advance.
--
View this message in context:
http://octave.1599824.n4.nabble.com/memory-exhausted-or-requested-size-too-large-for-range-of-Octave-s-index-type-tp3927828p3927828.html
Sent from the Octave - General mailing list archive at Nabble.com.
_______________________________________________
Help-octave mailing list
address@hidden
https://mailman.cae.wisc.edu/listinfo/help-octave
I'll help point you in the right direction. All comments assume that by binary
format you mean you want a binary formatted file. You will want to read and
write in memory and file output size efficient chunks to avoid running out of
memory and to minimize how long it takes to write. The write size should be
optimized to the output buffer size. This implies a loop, possibly an embedded
write loop inside of a large read, or vice versa. Note, Octave is not super
efficient at loops and reading 4.5 GB of data in a loop is going to take
awhile. You might be better off using a more appropriate programming language
if all you intend to do with the data is convert ascii to binary.
There are a number of read/write options in Octave.
To query command syntax use:
help <command_name>
To get more information and cross-linked commands
doc <command_name>
Doc has the advantage of a see also section at the end of the documentation.
Commands you may want to look up:
csvread -- you will have to read in row limited incremental chunks
dlmread
fread
fwrite
dlmwrite
If you want to process the data investigate reduced precision storage if your
data allows it, ie uint16 or smaller to minimize the memory footprint. Though
even this may not be enough to allow you to manipulate the whole set
efficiently.
William Krekeler