RE: memory exhausted or requested size too large for range of Octave's index type

[William Krekeler]

-----Original Message-----
From: address@hidden [mailto:address@hidden On Behalf Of Ganesh7789
Sent: Saturday, October 22, 2011 4:02 AM
To: address@hidden
Subject: memory exhausted or requested size too large for range of Octave's 
index type

I'm new to this. I was trying to read 4.5GiB file which is in csv format. But
my RAM is 3GiB. I'm runnig fedora 15. I have core i5 first generation. I
wanted to convert it to binary format. Can anyone tell me how to do that
using Octave?
Thanks in advance.

--
View this message in context: 
http://octave.1599824.n4.nabble.com/memory-exhausted-or-requested-size-too-large-for-range-of-Octave-s-index-type-tp3927828p3927828.html
Sent from the Octave - General mailing list archive at Nabble.com.
_______________________________________________
Help-octave mailing list
address@hidden
https://mailman.cae.wisc.edu/listinfo/help-octave

I'll help point you in the right direction. All comments assume that by binary 
format you mean you want a binary formatted file. You will want to read and 
write in memory and file output size efficient chunks to avoid running out of 
memory and to minimize how long it takes to write. The write size should be 
optimized to the output buffer size. This implies a loop, possibly an embedded 
write loop inside of a large read, or vice versa. Note, Octave is not super 
efficient at loops and reading 4.5 GB of data in a loop is going to take 
awhile. You might be better off using a more appropriate programming language 
if all you intend to do with the data is convert ascii to binary.

There are a number of read/write options in Octave.

To query command syntax use:
        help <command_name>
To get more information and cross-linked commands
        doc <command_name>

Doc has the advantage of a see also section at the end of the documentation.

Commands you may want to look up:
csvread -- you will have to read in row limited incremental chunks
dlmread
fread
fwrite
dlmwrite

If you want to process the data investigate reduced precision storage if your 
data allows it, ie uint16 or smaller to minimize the memory footprint. Though 
even this may not be enough to allow you to manipulate the whole set 
efficiently.

William Krekeler