[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Reading array slow ( and dlmread bug )
From: |
Paul Laub |
Subject: |
Re: Reading array slow ( and dlmread bug ) |
Date: |
Mon, 21 Feb 2005 08:31:52 -0800 |
Peter,
If speed of reading and writing data is critical to you, then you
might consider writing an external C or Perl program converting your
data into Octave's own binary format. Binary format data loads
extremely fast. Or, more simply, load your ASCII data files once into
Octave and then export each of them in binary. Use the "save -binary"
command. Then every subsequent loading of your data will be fast.
Paul Laub
On Mon, 21 Feb 2005 10:54:48 +0000, peter <address@hidden> wrote:
>
> I tried a new construct this morning,
> however it is still a factor of 10 slower
> than the equivalent perl.
>
> rdiff_2.m
> --------------------------------
> #!/usr/bin/octave -q
> fid = fopen("new.out","rt") ;
> while( (line = fgetl(fid)) != -1 )
> ;
> endwhile
>
> Peter
>
> peter wrote:
>
> >
> > I am writing a small utility program to compare
> > files containing numerical data.
> >
> > I found that reading the files (with octave) was very slow
> > compared to perl. The format of the data files is shown
> > below.
> >
> > 38157 10872 33592 26089 47591 14720 45843 16639
> > .....
> > ......
> > 55002 29660 45918 52316 36063 56511 31388
> > 36950
> > I have enclosed a small function to generate
> > the data (generate.m)
> >
> > I tried to use "load" and "dlmread". The execution speed
> > is show below (slightly edited output).
> >
> > Sun Feb 20 18:40:50 GMT 2005
> > octave dlmread
> > ans = 100000 11 <=== incorrect size of array returned by "dlmread"
> >
> > Sun Feb 20 18:40:56 GMT 2005
> > Octave load
> > ans = 100000 10
> >
> > Sun Feb 20 18:41:17 GMT 2005
> > perl
> > Sun Feb 20 18:41:18 GMT 2005
> >
> > In this case "dlmread" took 6 seconds and "load" took 21
> > seconds. The perl program took 1 second .....
> >
> > I think there is a bug in "dlmread" as empty space followed
> > by "\n" is seen as a zero value. The size for the array is therefore
> > reported incorrectly.
> >
> > I think the difference is remarkable, given that
> > the octave program is essentialy executing "C"
> > code.
> >
> > Please find enclosed the code snippets I used.
> >
> > rdiff.sh
> > -----------------------
> > #!/bin/bash
> > date
> > echo "octave dlmread"
> > ./rdiff.m
> > date
> > echo "Octave load"
> > ./rdiff_1.m
> > date
> > echo "perl"
> > ./rdiff.pl
> > date
> >
> > rdiff.m
> > ---------------------
> > #!/usr/bin/octave -q
> > a = dlmread("new.out"," ") ;
> > size(a)
> >
> > rdiff_1.m
> > --------------------
> > #!/usr/bin/octave -q
> > a = load("new.out") ;
> > size(a)
> >
> > rdiff.pl.
> > -------------------------
> > #!/usr/bin/perl
> > open(fid,"new.out");
> >
> > while($record = <fid>) {
> > @number = split(/[ \t]+/,$record) ;
> > #print("@number\n");
> > }
> > close(fid);
> >
> >
> > generate.m
> > -------------------------------
> > #!/usr/bin/octave -q
> >
> > 1 ;
> >
> > function save_array(name,x)
> > fid = fopen(name,"wt") ;
> > for n=1:rows(x)
> > fprintf(fid,"%8d ",x(n,:)) ;
> > fprintf(fid,"\n") ;
> > endfor
> > fclose(fid) ;
> > endfunction
> >
> > N = 100000 ;
> > a = round(rand(N,10)*2^16) ;
> > save_array("new.out",a) ;
> >
> >
> >
> > -------------------------------------------------------------
> > Octave is freely available under the terms of the GNU GPL.
> >
> > Octave's home on the web: http://www.octave.org
> > How to fund new projects: http://www.octave.org/funding.html
> > Subscription information: http://www.octave.org/archive.html
> > -------------------------------------------------------------
> >
> >
>
> -------------------------------------------------------------
> Octave is freely available under the terms of the GNU GPL.
>
> Octave's home on the web: http://www.octave.org
> How to fund new projects: http://www.octave.org/funding.html
> Subscription information: http://www.octave.org/archive.html
> -------------------------------------------------------------
>
>
-------------------------------------------------------------
Octave is freely available under the terms of the GNU GPL.
Octave's home on the web: http://www.octave.org
How to fund new projects: http://www.octave.org/funding.html
Subscription information: http://www.octave.org/archive.html
-------------------------------------------------------------