Octave performance slowdown over time

[Rik]

4/28/17

This has been discussed before, but I thought I'd mention it again as there was a question about Octave's slow loop performance.

Ideally, Octave would have a performance benchmark of some sort that we could run after significant code changes to make sure that there has not been a performance regression.  This is is analogous to the BIST tests in specific functions and the test suite in the test/ directory which are run after major code changes to make sure we haven't mistakenly altered Octave's functional behavior.  But, we don't have that yet so I've been using a simple double for loop as a benchmark as this is known to be a problematic corner case.

-- Test Code --
N = 102;

bm = zeros (N,1);

for cnt = 1:N
  a = 1; b = 1;
  tic;
  for i=1:1000; for j=1:1000; a = a + b + 1.0; end; end;
  bm(cnt) = toc;
endfor

## Remove outliers
[tmp,idx] = max (bm);
bm(idx) = [];
[tmp,idx] = min (bm);
bm(idx) = [];

printf ("Median: %s", disp (median (bm)));
printf ("Mean: %s", disp (mean (bm)));
printf ("STD: %s", disp (std (bm)));
-- End Test Code --

I happened to upgrade my OS in the last two weeks and I thereby lost all my historical Octave binaries.  I downloaded the source code for various old versions and, with some tweaking of code and configure options, was able to compile significant releases from 3.0.5 to the present.  Running the benchmark code gives the following results:

Version	3.0.5	3.2.4	3.4.3	3.6.4	3.8.2	4.0.3	4.2.1	4.2.1 (GUI)	dev
Median:	1.0665	0.71457	0.57382	0.73447	0.81407	1.4794	1.4263	1.677	1.4072
Mean:	1.0915	0.73668	0.57882	0.74629	0.82407	1.4869	1.439	1.6886	1.425
STD:	0.10208	0.065424	0.019963	0.034051	0.032718	0.040534	0.045276	0.044694	0.047039

There was a huge step change from 3.8.2 to 4.0.3.  Using the GUI is also ~20% slower than running the CLI version.

--Rik