On Wed, Jun 22, 2011 at 12:31 PM, John W. Eaton
<address@hidden> wrote:
On 22-Jun-2011, Ole Jacob Hagen wrote:
| I am using Octave for all my scientific work.
| But I have also faced some problems regarding performance in file loading.
|
| I have data files, that are 500 - 1000 MB in size, with 133 columns of data.
|
| I am loading a log file (500 MB) and then creating up to 10 plots of them.
| Here is the performance:
|
| * Linux: File loading + plotting: 3 minutes
|
| * Windows XP(MinGw): File loading + plotting: 30-40 minutes.
|
| File loading operation eg data = "" is using 80-90%
| of total time.
|
| Can a pure MSVC build increase performance of data files with that size,
Thinking that the problem must be MinGW and that everything would be
fixed if using MSVC seems like jumping to conclusions to me.
As someone else asked, is this on the same hardware?
Even 3 minutes to load a 1GB file seems like a long time to me. It's
pretty well known that loading text files in Octave is not very
efficient. It would be great if someone could do some work to
optimize it.
I wonder if it could be related to line-ending translations? Does the log file have Unix/DOS/Mac line endings? You could try changing the line endings of the file and see if the performance improves.
--judd