Re: Need a faster way of reading files.

[Jordi Gutiérrez Hermoso]

On 2 June 2010 10:23, Ole Jacob Hagen <address@hidden> wrote:

> I am using data = load('mydata.txt'), but this is very very slow,
> compared to SciPy. SciPy uses one minute, while Octave is using
> approx 5-10 minutes. Haven't measured time consumption, but can come
> back with that one. ;-)
[snip]
> How can I accelerate my data loading into memory in Octave?

Octave looks in the header comments for information about the size of
your data. If it doesn't find any, it has to do two passes through
your data to figure out its size. You might get some speedup if you
put that information in your data so Octave only needs to do one pass.
Save a sample matrix and look at the headers to see what Octave
expects.

I'm curious what SciPy is doing, though... can you benchmark its
memory consumption while it's doing this? If it's using a lot more
memory than Octave, I can imagine what sort of thing it's doing in
exchange for speed. I also wonder if it's doing as many checks as
Octave (NaNs, infs, various ways of formatting the data, etc).