Re: Need a faster way of reading files.

[Bård Skaflestad]

On Wed, 2010-06-02 at 17:23 +0200, Ole Jacob Hagen wrote:
> Hi, 
> 
> I have large data files, with 90 columns. Typical size is 600-700 MB.
>  Data is stored in ascii format. This is data collected for 7200 s,
>  with sampling interval 100 Hz. 
> 
> I am using data = load('mydata.txt'), but this is very very slow,
>  compared to SciPy. SciPy uses one minute, while Octave is using approx
>  5-10 minutes. Haven't measured time consumption, but can come back
>  with that one. ;-) 
> 
> I've tried to use fread, but the data is not recognizable at all. I
>  know how many columns I have in my data file. 
> 
> How can I accelerate my data loading into memory in Octave?

Since you're using LOAD, I'll assume that the values are floating point
numbers.  This is a shot in the dark, but have you tried FSCANF?  Note
that FREAD is (mainly?) for binary data.  Would you mind trying the
following statements

        fid = fopen('mydata.txt', 'rt')
        tic; table = fscanf(fid, '%f', [90, inf]).'; toc

and reporting back on the tic/toc pair (as well as if the reading was
successful or not)?



Best regards,
-- 
Bård Skaflestad                            <address@hidden>
SINTEF ICT, Applied mathematics