[Octave-bug-tracker] [bug #51871] loading '-ascii' format files is slow

[Dan Sebald]

Follow-up Comment #21, bug #51871 (project octave):

Perhaps the easiest approach would be to avoid the standard functions like
getline, strings, etc., and instead write a little C routine that

1) Identifies what the new-line character is
2) Determines what the longest line of the file is

With that information, we can pre-assign a buffer length in which to capture
at least the longest line.

Something like (psuedo code):


char *newlineopts[] = {
  "\cr";
  "\cr\lf";
  "\r";
}

maxstrlen = 0;
thisstrlen = 0;
newlinestr = \null;
OPEN STREAM FOR CHARACTER READING
while (!oef) {
  GET CHAR cval FROM STREAM
  for (i=0; i<length(newlineopts); i++) {
    if (newlineopts[i][0] == cval) {
      if (newlineopts[i][1] == '\0') {
        newlinestr = newlineopts[i];
        maxstrlen = thisstrlen > maxstrlen ? thisstrlen : maxstrlen;
        thisstrlen = 0;
        continue;
      } else {
        GET CHAR cval FROM STREAM;
        if (newlineopts[i][1] == cval) {
          newlinestr = newlineopts[i];
          maxstrlen = thisstrlen > maxstrlen ? thisstrlen : maxstrlen;
          thisstrlen = 0;
          continue;
        }
      }
    }
  thisstrlen++;
}

// We now know the maximum line length in the file and what
// the EOL sequence is.
ASSIGN BUFFER linebuf OF SIZE maxstrlen+1.
REWIND FILE STREAM
while (!eof) {
  FILL linebuf UNTIL NEW LINE SEQUENCE FOUND
  PROCESS WITH THE CURRENT CODE
}


Something like that.  I'm just trying to make the point that we can probably
create a custom "getline" that is very efficient without too much effort.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?51871>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/