[Octave-bug-tracker] [bug #51871] loading '-ascii' format files is slow

[Dan Sebald]

Follow-up Comment #40, bug #51871 (project octave):

Here's a reference that compares various file input methods:

https://stackoverflow.com/questions/18688763/why-is-istream-ostream-slow

Anything that brings in one individual character at a time is slow.  However,
note that the person's example is kind of simplified and suited for reading
large hunks of data at once.  That is, it's not geared to reading a line of
ascii text, then another, then another, then another.

I've implemented some of the methods I outlined in previous posts.  They
aren't as slow as the current code, but they are about 50% slower than
speed-up-load-ascii-v5.patch (which uses getline).

I have an idea though.  What if I write getline_alleol() to bring in a hunk of
data?  One asks, How do we know the size of the hunk to bring in?  If we read
too much, then seek back to the EOL (0x0A, 0x0D, 0x0D/0x0A), that becomes
inefficient.  Too short and that becomes inefficient as well.  How about
choosing the hunk size to be the length of the last line read?  That should be
a good initial guess.  So long as the data is fairly consistent in its field
width from line to line, this should be efficient.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?51871>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/