Re: [gnugo-devel] 100 game series

[Arend Bayer]

On Wed, 20 Feb 2002, Daniel Bump wrote:

> > If we want to play such matches regularly, it might make sense to have a
> > command line option to turn off all score-dependent behaviour. Then the
> > average score would be a meaningful information as well.
> 
> Score-dependent behavior consists of playing more cautiously
> when ahead. This hopefully has the effect of increasing the
> win rate though decreasing the margin. I'm not sure I believe
> that it's more meaningful to turn this off.

I don't exactly mean that the score is more meaningful; but it should
be a easier (i.e., fewer games suffice) to collect statistically significant
data about the average score than about the winning percentage.

To justify my claim let's take your 100 game series.
I get an average score of ~ W+15.3 +- 2.8. (That suggests 1.5 grades, btw.)
For 68 wins out of 100 games I get a standard deviation of +- 4.6%.

So the score determines the better player by 5.5 standard deviations,
the winning percentage by 4.
Now, of course, it is irrelevant whether we know the better player with a
certainty of 98 or 99.9%. However, imagine a patch that improves GNU Go's
play by 5 pts per game on average. Given the standard deviation from
above of +-28 pts per game, this would mean a winning probability of
appr. 57%.  If you want to be sure by 95% (3 standard deviations IIRC)
  which is the better
play you would need
 - appr. 280 games if you use the scores (5 +- 28 / \sqrt{280})
 - appr. 420 games if you only use results (57% +- 48% / \sqrt{420})
  
Ok, not such a big difference, I admit.

Arend