Réf. : Re: JSD question [WAS : [Bug-gnubg] GNU Q uestion]

[Massimiliano . Maini]

>> Just another detail : if you compute the jsd as you said, you implicitly
>> assume the the two equity distributions (assumed normal) are independent.
>> In such a case Var(X-Y) = Var(X)+Var(Y), and the computation of the stderr
>> described above is fine.
>
>Yes the assumption being made is that you run enough trials that the
>result approximates a normal distribution and that the results of the
>trials are independant.
>
>In a sense I have simply automated what many users of backgammon
>programs already do, which is to see if the equities given by the
>rollout are far enough apart that even with the standard error being
>reported the result is still probably correct.
>
>For any given rollout, it's hard to say how many trials will be
>required (if it ever even occurs) such that the result will be a
>normal distribution. I think that the rollouts of different moves are
>independant with the main problem area being if gnubg has systematic
>errors handling some particular feature of a rollout.

In fact the possibility of the rollouts equity not being independant
scares me more than the possibility of them being non-normal : I do think
that they are "reasonably normal" for a fair number of trail, but if a
correlation exists, a higher number of trial will only point out the
correlation with a better precision.

As an example, imagine that in the hypothetical game tree you start from
your initial position and you consider only two branches (the 2 moves you
want to rollout). It may happen that the two sub-trees are not disjoint
in early stages and in this case, there may be an important correlation
between the two equity ditributions.
(Note that the trees will always share something, at least in a full rollout
at the end, when bearing off, but this may be numerically irrelevant).

>> But I'm not sure this independency assumption is reasonable : I would
>> compute the estimate of the stderr of the difference of the two equity
>> by it's definition (just like you compute the estimates of the stderr
>> of the equities, but on the equity difference). The estimate would then
>> include eventual correlation terms and would be equivalent (for a number
>> of samples large enough) to the computation whenever the distributions
>> were really independent.
>
>Are you suggesting treating a rollout of two different moves for n trials
>as n pairs of results (or n occurrences of a value for the equity
>difference) which would then be treated as an evaluation on it's own?

Yes : n occurrences of the equity difference, sum the squares, divide by n
and take the square root (classical stderr estimator).

>This could be done, although for a rollout of more than two options,
>you'd want to be keeping the equity differences for every pair of
>options, as you don't know which one will be the 'best' which will be
>used as the norm for all the comparisions (few people will care about
>the relative ranking of the 3rd and 4th best plays, only the ranking
>vs. the best play).

Right.

>> - eventually (if anyone else shares my doubts), run some test
>
>I think it's not to hard to do the above (assuming that's what you're
>suggesting) the hardest part is probably finding a way to display the
>results.

Why ? Just show the new "estimated jsd" (e-jsd) between the currently #1
move and all the others, just like what it is done for the currently
imlemented "assuming independancy jsd" (ai-jsd).

At the end, you'll just need to compare the M-1 ai-jsd with the M-1
e-jsd (for a rollout of M moves) and see if they seem to agree or not.


MaX.


P.S.
I may be wrong about all this (I haven't been playing with this stat stuff
for a while), so I would wait for confirmation before any software effort
(I've seen the CC to bug-gnubg but I haven't received anything from it
this afternoon).