Re: Fast generation of text file

[Andy Buckle]

On Wed, Jun 1, 2011 at 9:11 AM, wild.manul <address@hidden> wrote:
> Hello
>
> I need to write task for linear programming solver in text file from Octave.
> The structure of text file is following:
>
> a11 x1+ a12 x2+ a13 x3+ ... +a1n xn
> ....
> an1 x1+ an2 x2+ an3 x3+ ... +ann xn
>
> where a11, ..., ann are given coefficients. x1,...,xn are symbols x with
> indexes. Coefficients and indexes are stored in matrix a:
> a11 1 a12 2 a13 3 ... a1n n
> ....
> an1 1 an2 2 an3 3 ... ann n
> Problem is that matrix is rather huge - 2000x2000 elements and writing it to
> text file is slow.
>
> I wrote the following code to do that:
>
> for i:row do
>  fprintf(fid,'%7.5f x%d+',a(i,:));
> end
>
> May be there is faster way how to do it?

I could not figure out a way to make it faster. Some notes, and thoughts:
- You do not have linebreaks in your fprintf.
- The looping through fprintf calls can be avoided, but this does not
help much. I don't know how to have linebreaks if the looping is
avoided.
- I tried writing the whole file with one fwrite call, instead of
fprintf. i did not include this in the example below. This actually
made it slightly slower.
- Use a local drive, rather than a network drive.
- An oct file may make things faster.

my output (for 500x500) on win xp mingw octave 3.2.4 from sf.net:
                                 no loop: 3.0s
                                    loop: 3.2s
                             make string: 10.5s
     make string (less memory thrashing): 3.2s

my script:

% generate some dummy data
n=500;
inds=repmat(1:n,n,1);
coeffs=rand(n);
a=zeros(n,2*n);
a(:,2:2:end)=inds;
a(:,1:2:end)=coeffs;

% write in one call to fprintf
tic
fid=fopen("example1.txt",'w');
fprintf(fid,'%7.5f x%d+',a');
fclose(fid);
printf("%40s: %.1fs\n","no loop",toc);

% loop and call fprintf
tic
fid=fopen("example2.txt",'w');
for i=1:rows(a)
  fprintf(fid,'%7.5f x%d+',a(i,:));
end
fclose(fid);
printf("%40s: %.1fs\n","loop",toc);

% construct string in memory first
tic
s="";
for i=1:rows(a)
  s=[s sprintf('%7.5f x%d+',a(i,:))];
end
fid=fopen("example3.txt",'w');
fprintf(fid,"%s",s);
fclose(fid);
printf("%40s: %.1fs\n","make string",toc);

% construct string in memory first: less memory thrashing
chunksize=2^22;
tic
s=char(zeros(1,chunksize));
usedlength=0;
for i=1:rows(a)
  append=sprintf('%7.5f x%d+',a(i,:));
  len=length(append);
  if length(s)<(usedlength+len)
        s=[s char(zeros(1,chunksize))]
  end
  s(usedlength+1:usedlength+len)=append;
  usedlength+=len;
end
fid=fopen("example4.txt",'w');
fprintf(fid,"%s",s(1:usedlength));
fclose(fid);
printf("%40s: %.1fs\n","make string (less memory thrashing)",toc);


-- 
/* andy buckle */