Re: Benchmarking with --enable-atomic-refcount

[Rik]

On 01/11/2014 11:33 PM, Daniel J Sebald wrote:
> On 01/10/2014 04:14 PM, Rik wrote:
>> 1/10/14
>>
>> John,
>>
>> I get a different, and slower performing, result with atomic reference
>> counting.
>>
>> I built the dev branch with simple flags (-O2) and with/without
>> --enable-atomic-refcount.  I disabled frequency scaling in the processor so
>> each test was running at a fixed frequency and the comparison would be
>> fair.  I also ran each test 3 times to see if there was any variability
>> (not much).  Using 'time make check' as the actual benchmark I got the
>> following:
>>
>> Without atomic-refcount
>> 276.757u 33.510s 5:31.48 93.5%  0+0k 123608+21184io 567pf+0w
>> 275.973u 33.010s 5:21.02 96.2%  0+0k 280+21056io 0pf+0w
>> 276.189u 32.442s 5:20.39 96.3%  0+0k 0+21040io 0pf+0w
>>
>> Mean of User+System: 309.29
>>
>> With atomic-refcount
>> 313.647u 34.046s 6:00.00 96.5%  0+0k 0+21104io 0pf+0w
>> 314.567u 33.998s 6:00.85 96.5%  0+0k 0+21072io 0pf+0w
>> 311.943u 32.382s 5:55.87 96.7%  0+0k 152+21080io 0pf+0w
>>
>> Mean of User+System: 346.86
>>
>> % change: +12.1%
>>
>> Since we have conflicting results maybe somebody else needs to run the
>> tests as well.  I ran 'make clean' in between re-configuring Octave.  It
>> shouldn't be necessary, but you never know what kind of cruft accumulates.
>
> Is make-check a good benchmark?  Make-check might be just occasionally
> touching on some operation that is significantly affected by atomic
> reference counts.  That is, there might be a certain type of memory
> manipulation, looping, or whatnot that is really slowed.  It might make
> sense to imagine a worst case scenario and devise a test around that.
>
> On the other hand, I do know that just the simplest of commands will
> cause a lot of manipulations of the ref count.  When I analyzed the mutex
> concept a year or so ago, I placed print statements at every location ref
> count is modified.  The value changes a lot.

I think this is an okay benchmark.  In general I think we want something
close to the average load rather than the worst case.  If atomic reference
counts slowed down only a tiny fraction of Octave, that most users never
experienced, then that would probably be acceptable.  Using a worst case
analysis wouldn't show us that this particular tradeoff is possible.  Right
now the test suite is the closest thing we have to an "average" experience
of running large portions of Octave's code base.

Also, as jwe mentioned, and you have pointed out as well, there are solid
conceptual reasons for believing that atomic reference counts will have a
pervasive, systemic effect.  This seems to be the case.

--Rik
>
> Dan
>
>
>