RE: [Bug-gnubg] Re: The importance of METs

[Albert Silver]

> > I tried  100 5pt 0ply matches woolsey.xml  agains snowie.xml.
Analysing
> > with  woolsey at  0ply a  rating discrepancy  of 2193  +-8  resulted
(as
> > estimated by the "error rate"). Again totally negligible.
> 
> Using error rates and the accompanying rating estimates will be
biased,
> so while it certainly looks like an negligible difference, we need to
> use luck adjustment and run a gazillion matches in order to establish
> the best MET.

Thanks for the results. I'd like to reiterate my suggestion of testing
the METs specifically at the scores where they differ the most. Zare
pointed out that the Woolsey table at 4-away 3-away looked completely
wrong compared to the Snowie table for example. Many othger scores may
be quite correct so testing all the scores may dilute the differences
one would feel. How about trying a series of match games where the score
is specifically 4-away 3-away and see how big a difference it makes?

Perhaps, as seems the case, the preference of one MET over another
doesn't make a big difference when including all scores, but perhaps
that difference is more meaningful at specific scores which is where the
new MET would be most beneficial. 
 
> If would be interesting to perform such an experiment at different
match
> scores, but what would be the value of a result? Will the result
depend
> on the bot used for the experiment? I could imagine that gnubg would
> prefer a gnubg-MET and snowie a snowie-MET, i.e., match equity table
> calculated based on rollouts (Ian's experiments).

The same thought had occurred to me, since one MET might better reflect
the bot's play and evaluations. Unfortunately it is impossible to test
as Snowie doesn't allow one to choose one's MET.

                                        Albert


> 
> Jørn