[Bug-gnubg] The race training and benchmark datasets

[Øystein Schønning-Johansen]

Hi, all!

Out of curiosity I've just started to look into the race datasets. I notice something really strange.

I take all the positions from http://files.gnubg.com/media/nn-training/pmichel/training_data/race-train-data.bz2 and from this I create a dataset for training.

Then I take all positions from the benchmark dataset.

http://files.gnubg.com/media/nn-training/pmichel/benchmarks/train.bm-1.00.bz2. and generate a validation training set.

(Of course I remove any position duplicated in the two datasets, such that the training and validation set are disjoint.)

I train a neural network. If I validate the training with a 10% fraction of the training dataset itself, I get a MSE error of about 1.0e-04. But if I validate against the dataset generated from train.bm-1.00.bz2 I get an MSE error of 7e-04. About 7 times higher!

This makes me believe that the rolled out positions in the race-train-data file is rolled out in an other way (different tool, different settings, different neural net?) than the positions in train.bm-1.00.bz2.

Joseph? Philippe? Ian? Others? Do you know how these data where generated? Is it maybe worth rolling these positions out again? I do remember that Joseph made a separate rollout tool, but I'm not sure what Philippe did?

(I also remember that the original benchmark was move based, and it calculates the loss based on incorrect moves picked, and that it might not be that interesting if the rollout values are abit wrong....)  

Best regards,

-Øystein