For loop benchmarks between Octave versions

[Rik]

6/6/14

All,

I followed up one of the recent posts about differing performance between
Octave versions.  Alas, I think it is quite true.  We don't yet have 100%
test coverage for behavior, but it seems like another wish list item would
be a set of benchmarking tests so that we don't accidentally introduce
large performance regressions.

The test code in question is a double for loop.  We expect for loops to be
inefficient and slow, but at least they shouldn't get any worse than they
already are.

--- Test Code ---
a = 1; b = 1; tic; for i=1:1000; for j=1:1000; a = a + b; end; end; toc
--- End Test Code ---

I fixed the CPU frequency before running the tests to avoid differences due
to frequency scaling.  I experimented a bit with taskset to ensure the
Octave process stayed on a single core, but it didn't make a difference so
I stopped using it (I would expect this since the Linux kernel also tries
for performance reasons not to shift running jobs off of a core).  All of
the versions are locally compiled on a Linux machine using roughly the same
options of '-O2'.

Results:

Version    3.2.4    3.4.3    3.6.4    3.8.0    3.8.1    gui-release
(no-gui-libs)    dev (no-gui-libs)    dev (gui)

     2.24418    1.71425    2.95948    3.4683    3.15712    6.50268  
 7.00682    8.09868
    2.24436    1.71648    2.97603    3.43708    3.15531    6.47645  
 6.93558    8.0038
    2.25401    1.7231    2.96663    3.46252    3.19536    6.46978  
 6.94941    8.07056
                                
Mean    2.24752    1.71794    2.96738    3.45597    3.16926    6.48297  
 6.96394    8.05768

There was a doubling in execution time from 3.4.3 to 3.6.4, mostly
unchanged from 3.6.4 to 3.8.1, and then another doubling on the gui-release
branch.  The gui-release branch and other releases going forward is
compiled with atomic refcounting which is necessary for the GUI, but does
have a performance penalty.  I'm re-compiling the gui-release branch with
atomic refcounting disabled to see how much it improves things, but I don't
have high hopes.  The last time I benchmarked this it was only a 10%
difference, rather than the 100% difference that the tests show.

--Rik