Re: Building on OSX without package managers

[Richard Campbell]

On Jan 25, 2011, at 12:17 AM, John W. Eaton wrote:

> On 24-Jan-2011, Richard Campbell wrote:
> 
> | I have 12 GB of RAM and I intend to use this on other machines with
> | 16+ GB. Today I'm trying to load a contiguous block of 920 MB of
> | complex doubles written to disk from some C code. Sometimes it
> | works, sometimes it gives me an out of memory error (the kind where
> | you have to restart Octave and try again).
> 
> Precisely how are you trying to load the data?  Is it written in a
> binary or text format?  If text, you can greatly improve Octave's
> performance for reading if you write a header like the one produced by
> Octave's "save -text" command so that you can use load to read the
> file and Octave won't have to guess the array size and possibly
> reallocate repeatedly to read the data.
> 
> If you are reading a text file, is the 920MB the size of the text
> file, or do you mean that you have that 920MB/16 ~= 115e6 complex
> array elements?
> 
> If the array is 115e6 elements, that is not large enough to need 64-bit
> indexing unless you generate much larger temporary values while you are
> working with your data.  But with Octave (or Matlab), if you do
> anything with the entire array, then you can expect to need
> generate some temporaries that are the same size as the original
> array.  But with current versions of Octave (3.3.x and later) the
> limit on the total amount of data is much greater than 2GB if you have
> a system with 64-bit pointers, even if you are building Octave to use
> 32-bit indexing (see the info in the manual about building with
> --enable-64 for more details).
> 
> jwe

It's not text. The binary block in question was written using a single in C 
along the lines of fwrite(ptr, sizeof(double complex), N, fid);

Here's what I'm doing in Octave:

raw = fread(fid, 2 * C, "double");
mat = complex(raw(1:2:end), raw(2:2:end));
mat = reshape(mat, dims);