Re: Graphics and memory opinions

[Daniel J Sebald]

John W. Eaton wrote:
> On 14-Dec-2008, Jordi Gutiérrez Hermoso wrote:
>
> | Right, std::list triples the storage size. Clearly std::deque or
>
> I don't see how a deque helps, it is just another variant of a list, 
> with overhead for each element.
>
> | std::vector are the ones to use here.
>
> Then you will just have to copy them into an Octave Array<> object. 
> So why not just use that to begin with?  If you are going to use an 
> array/vector object then trick is efficiently approximating the the 
> size of the matrix so you can minimize the number of times you have
> to resize and copy.
>
> | Give me a clue on what function I should play with... It is |
> read_ascii_data(...), right?
>
> Look at the functions in ls-mat-ascii.cc.

Yeah, this can't be very fast.  Things are done one character at a time.  
That's taking the object oriented paradigm to too granular of a level.  (OOP is 
great, but...)  In particular

get_mat_data_input_line (std::istream& is)
{
   std::string retval;
...
   while (is.get (c))
...
              retval += c;
...
}

First, that calls a function to get every single character (slow).  Second, 
every time the function is called a new instance of a std::string is created 
(slow).  Third, that string object is empty and then every character is added 
one at a time (very slow); depending on how the string library is implemented, 
that could mean the string has to assign new memory one character bigger and 
then free the previous memory (perhaps it is more efficient than that, but my 
guess is that it isn't).

There is more.  Then, yes, as Jordi says, first the number of rows is 
determined:

 get_lines_and_columns (is, filename, nr, nc);

after which the matrix is assigned:

 Matrix tmp (nr, nc);

(is that on some kind of stack?)


My suggestion for this is more C-like and less object-like.  Bring in the file 
contents, say, 32 KB at a time and keep track of the index into the buffer.  
Use the efficient string functions if possible to find characters.  To avoid 
looping through the file twice, make the size of Matrix tmp dynamic so that it 
always has enough room to contain the lines that have been read.  E.g., as 
lines are read in, its size would be

tmp(2, nc)
tmp(4, nc)
tmp(8, nc)
tmp(16, nc)
tmp(32, nc)
tmp(64, nc)
tmp(128, nc)
tmp(254, nc)
tmp(512, nc)

etc.  This generally works out to be very efficient and never wastes too much 
memory.

There is probably 16 hrs worth of work to transform ls-mat-ascii.cc into 
something efficient and tested for no bugs and leaks.

Dan