[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [gnugo-devel] auto generated regression tests
From: |
Arend Bayer |
Subject: |
Re: [gnugo-devel] auto generated regression tests |
Date: |
Wed, 14 Aug 2002 16:36:57 +0200 (CEST) |
On Sat, 3 Aug 2002, Evan Berggren Daniel wrote:
> attached are a collection of games and regression tests. They all are
> based on matcher_status inconsistencies from the matcher_check.pl script.
> Expected passes are marked relative to gnugo 3.2, however gnugo 3.3.5
> gains at least one test. Some of these I have analyzed and put in checks
> on owl_attack, etc, but most are just based on dragon_status. I'm not
> sure all the tests are correct (I may have read things wrong), so someone
> else should probably look them over.
>
> Anyway, hope these help.
I have looked through the tests and I think they are pretty useful.
As a general comment, "dragon_status" takes a lot more time than
"owl_defend/attack". I am all in favour of adding a lot more of
these semi-automatic tests to the regression, but we don't want the
regressions run to take unnecessariyl longer, so I'd suggest to use the
latter. (I.e., instead of
X dragon_status XX
#? [!dead]
we can use
X owl_defend XX
#? [!0]
or even better
X owl_defend XX
#? [1 (list of working defense moves)]
There were two or three tests where I came to a different conclusion.
These changes, plus the conversion away from using dragon_status, are
in the attached .tgz-file as well as in the patch below.
As you have already pointed out in another e-mail, this type of self-play
catches a particular type of owl mistake. I would think that analyzing
games against humans in the same manner would be extremely useful, too,
and possibly catch a bigger variety of mistakes.
> Also, has anyone tried running the matcher_status script? does gnugo
> finish games completely when you do?
Did GNU Go play the passes at the "end" of auto010.sgf?? That would be
very weird, and it doesn't happen when I let GNU Go generate a move
there. Would be quite a bug.
Arend
diff -u gnugo-3.2/regression/auto01.tst autotests/regression/auto01.tst
--- gnugo-3.2/regression/auto01.tst Fri Aug 2 19:15:23 2002
+++ autotests/regression/auto01.tst Wed Aug 14 16:11:01 2002
@@ -1,59 +1,63 @@
#auto generated owl inconsistencies
loadsgf games/auto_gen/auto001.sgf 136
-1 dragon_status p11
-#? [!dead]*
+1 owl_defend p11
+#? [1 Q11]*
loadsgf games/auto_gen/auto001.sgf 139
-2 dragon_status r16
-#? [!dead]*
+2 owl_attack r16
+#? [1 (O11|M10)]*
loadsgf games/auto_gen/auto001.sgf 140
-3 dragon_status r16
-#? [!dead]*
+3 owl_defend r16
+#? [1 (O11|T15|T13)]*
loadsgf games/auto_gen/auto002.sgf 186
-4 dragon_status n8
-#? [!dead]*
+4 owl_defend n8
+#? [1 (R1|N3|N5|M5|N6)]*
loadsgf games/auto_gen/auto003.sgf 52
-5 dragon_status s9
-#? [!dead]*
+5 owl_defend s9
+#? [1 Q10]*
loadsgf games/auto_gen/auto004.sgf 9
-6 dragon_status f3
-#? [!dead]*
+6 dragon_defend f3
+#? [!0]*
-loadsgf games/auto_gen/auto005.sgf 29
-7 dragon_status f17
-#? [!dead]*
+# This is a duplication of the previous one. /ab
+# loadsgf games/auto_gen/auto005.sgf 29
+# 7 dragon_status f17
+# #? [!dead]*
loadsgf games/auto_gen/auto006.sgf 25
-8 dragon_status g4
-#? [!dead]*
+8 owl_defend g4
+#? [1 (H5|J4|G5)]*
-loadsgf games/auto_gen/auto006.sgf 78
-9 dragon_status m17
-#? [critical]*
-
-loadsgf games/auto_gen/auto006.sgf 82
-10 dragon_status m17
-#? [critical]*
+# I think M17 is dead. After White H18, black can e.g. play L18. /ab
+# loadsgf games/auto_gen/auto006.sgf 78
+# 9 dragon_status m17
+# #? [critical]*
+
+# M17 lookd dead. /ab
+# loadsgf games/auto_gen/auto006.sgf 82
+# 10 dragon_status m17
+# #? [critical]*
loadsgf games/auto_gen/auto006.sgf 81
11 owl_attack m17
-#? [j18]*
+#? [1 J18]*
loadsgf games/auto_gen/auto006.sgf 94
-12 dragon_status r12
-#? [!alive]*
+# This is duplicated by the next test.
+# 12 dragon_status r12
+# #? [!alive]*
+# P13 probably does not work. (P13-Q12-P12-Q9-Q11-Q7 and black should win
+# the semeai.) /ab
13 owl_attack s11
-#? [q12|q13|p13]*
-
-loadsgf games/auto_gen/auto006.sgf 121
-14 dragon_status s9
-#? [!alive]*
+#? [1 (Q12|Q13)]*
-15 owl_attack s9
-#? [!0]*
+# This is a difficult semeai problem.
+loadsgf games/auto_gen/auto006.sgf 99
+15 owl_analyze_semeai S9 S10
+#? [ALIVE DEAD S6]*
diff -u gnugo-3.2/regression/auto02.tst autotests/regression/auto02.tst
--- gnugo-3.2/regression/auto02.tst Sat Aug 3 00:40:49 2002
+++ autotests/regression/auto02.tst Wed Aug 14 16:14:37 2002
@@ -1,43 +1,45 @@
loadsgf games/auto_gen/auto007.sgf 20
-1 dragon_status q13
-#? [!dead]*
+1 owl_defend q13
+#? [1 (Q11|P12)]*
loadsgf games/auto_gen/auto007.sgf 42
-2 dragon_status q13
-#? [!dead]*
+2 owl_defend q13
+#? [!0]*
+# C16 is probably best. /ab
loadsgf games/auto_gen/auto008.sgf 58
-3 dragon_status d16
-#? [!dead]*
+3 owl_defend d16
+#? [1 (C16|D17|E15)]*
loadsgf games/auto_gen/auto008.sgf 150
-4 dragon_status c12
-#? [!alive]*
+4 owl_attack c12
+#? [1 (A14|A12)]*
+# Good problem for connection-related owl tuning.
loadsgf games/auto_gen/auto008.sgf 180
-5 dragon_status a15
-#? [!dead]*
+5 owl_defend a15
+#? [1 (B9|D9)]*
loadsgf games/auto_gen/auto009.sgf 94
-6 dragon_status f3
-#? [!dead]*
+6 owl_defend f3
+#? [!0]*
loadsgf games/auto_gen/auto010.sgf 66
-7 dragon_status q17
-#? [!alive]*
+7 owl_attack q17
+#? [1 (S15|R15)]*
loadsgf games/auto_gen/auto010.sgf 64
-8 dragon_status q17
-#? [!alive]*
+8 owl_attack q17
+#? [!0]*
loadsgf games/auto_gen/auto010.sgf 77
-9 dragon_status c18
-#? [!alive]*
+9 owl_attack c18
+#? [D18]*
loadsgf games/auto_gen/auto011.sgf 43
-10 dragon_status d9
-#? [!dead]*
+10 owl_defend d9
+#? [!0]*
loadsgf games/auto_gen/auto012.sgf 168
-11 dragon_status h15
-#? [!dead]*
+11 owl_attack h15
+#? [0]*
diff -u gnugo-3.2/regression/auto03.tst autotests/regression/auto03.tst
--- gnugo-3.2/regression/auto03.tst Sat Aug 3 04:59:24 2002
+++ autotests/regression/auto03.tst Wed Aug 14 15:45:25 2002
@@ -1,47 +1,50 @@
loadsgf games/auto_gen/auto013.sgf 47
-1 dragon_status h16
-#? [!dead]*
+1 owl_defend h16
+#? [!0]*
loadsgf games/auto_gen/auto013.sgf 51
-2 dragon_status h16
-#? [!dead]*
+2 owl_defend h16
+#? [!0]*
loadsgf games/auto_gen/auto013.sgf 53
-3 dragon_status l17
-#? [!alive]*
+3 owl_attack l17
+#? [K17]*
+# This is a connection problem. /ab
loadsgf games/auto_gen/auto014.sgf 106
-6 dragon_status l14
-#? [!unknown]*
+6 owl_attack l14
+#? [N12]*
loadsgf games/auto_gen/auto014.sgf 107
7 owl_defend l14
#? [n12]*
loadsgf games/auto_gen/auto015.sgf 145
-8 dragon_status t7
-#? [!alive]*
+8 attack t7
+#? [T5]*
loadsgf games/auto_gen/auto016.sgf 37
-9 dragon_status j15
-#? [!dead]*
+9 owl_defend j15
+#? [!0]*
loadsgf games/auto_gen/auto016.sgf 48
-10 dragon_status p16
-#? [!dead]*
+10 owl_defend p16
+#? [O16|P15]*
loadsgf games/auto_gen/auto016.sgf 53
-11 dragon_status q4
-#? [!dead]*
+11 owl_defend q4
+#? [P4|P3|Q3|R4]*
loadsgf games/auto_gen/auto016.sgf 55
-12 dragon_status n5
-#? [!dead]*
+12 owl_defend n5
+#? [!0]*
-loadsgf games/auto_gen/auto017.sgf 15
-13 dragon_status c13
-#? [!alive]*
+# I don't think black can kill C13. Of course, white should still answer
+# the black attachment, hence this might be worth a genmove test instead. /ab
+# loadsgf games/auto_gen/auto017.sgf 15
+# 13 dragon_status c13
+# #? [!alive]*
loadsgf games/auto_gen/auto017.sgf 45
-14 dragon_status q4
-#? [!dead]*
+14 owl_defend q4
+#? [!0]*
diff -u gnugo-3.2/regression/auto04.tst autotests/regression/auto04.tst
--- gnugo-3.2/regression/auto04.tst Sat Aug 3 20:11:45 2002
+++ autotests/regression/auto04.tst Wed Aug 14 15:55:05 2002
@@ -1,45 +1,27 @@
loadsgf games/auto_gen/auto018.sgf 103
-1 dragon_status r14
-#? [!alive]*
+1 owl_attack r14
+#? [!0]*
loadsgf games/auto_gen/auto019.sgf 76
-2 dragon_status e18
-#? [!dead]*
+2 owl_defend e18
+#? [!0]*
loadsgf games/auto_gen/auto020.sgf 67
-3 dragon_status n4
-#? [!dead]*
-
-4 owl_defend n4
-#? [m5]*
+3 owl_defend n4
+#? [M5|L4|N5|L5]*
loadsgf games/auto_gen/auto021.sgf 72
-5 dragon_status g15
-#? [dead]
-
-6 owl_defend g15
-#? [!j14]
-
-7 owl_does_defend j14 g15
+7 owl_defend j14
#? [0]*
loadsgf games/auto_gen/auto021.sgf 71
-8 dragon_status g15
-#? [critical]*
-
-9 owl_does_attack k15 g14
-#? [1]*
+8 owl_attack g15
+#? [K15]*
loadsgf games/auto_gen/auto022.sgf 37
-10 dragon_status e18
-#? [!dead]*
-
11 owl_defend e18
-#? [d17]*
+#? [D17|F12|G14|G18|H18]*
loadsgf games/auto_gen/auto022.sgf 136
12 owl_attack g4
-#? [d2]*
-
-13 dragon_status g4
-#? [critical]*
+#? [D2]*
Common subdirectories: gnugo-3.2/regression/games and autotests/regression/games
auto_tests.tgz
Description: GNU Unix tar archive
- Re: [gnugo-devel] auto generated regression tests, (continued)
- Re: [gnugo-devel] auto generated regression tests, bump, 2002/08/05
- Re: [gnugo-devel] auto generated regression tests, Gunnar Farneback, 2002/08/05
- Re: [gnugo-devel] auto generated regression tests, bump, 2002/08/05
- Re: [gnugo-devel] auto generated regression tests, bump, 2002/08/05
- Re: [gnugo-devel] auto generated regression tests, bump, 2002/08/05
- Re: [gnugo-devel] auto generated regression tests, Gunnar Farneback, 2002/08/05
- Re: [gnugo-devel] auto generated regression tests, bump, 2002/08/06
- Re: [gnugo-devel] auto generated regression tests, Gunnar Farneback, 2002/08/07
- Re: [gnugo-devel] auto generated regression tests, Gunnar Farneback, 2002/08/07
- Re: [gnugo-devel] auto generated regression tests, Gunnar Farneback, 2002/08/05
Re: [gnugo-devel] auto generated regression tests,
Arend Bayer <=
Re: [gnugo-devel] auto generated regression tests, bump, 2002/08/08