[gnugo-devel] Regression cleanup

[Trevor Morris]

Before I delve deeper into the test case cleanup, I'd like folks
opinion of the approach I'm taking:
 - Keep all reference comments in the *.tst files themselves as
   comments, instead of in separate files like STRATEGY, etc.
 - Begin using a standard notation to prioritize and categorize
   the test cases.  Notably, CATEGORY, DESCRIPTION, and SEVERITY
   fields.  See the patch below for examples.

You can see examples in the following patch.

You can also see the comments showing up in action by following
the strategy.tst failure links on the new and improved:
  http://www.public32.com/regress/
Most notably are the markers to indicate the bad and good moves,
as well as the category, description, and severity sections.

Here's the patch:
  http://www.public32.com/games/go/trevor_strat_1_10.3


Index: strategy.tst
===================================================================
RCS file: /cvsroot/gnugo/gnugo/regression/strategy.tst,v
retrieving revision 1.10
diff -r1.10 strategy.tst
52a53,61
> 
> #CATEGORY=BAD_PROBLEM
> #DESCRIPTION=Not clear enough.
> #SEVERITY=0
> # From STRATEGY:
> #    2.7.178 makes the move at P17 which is better than the game
> #    move at P18 but probably inferior to the recommended move at P15.
> #    It is not totally clear to me that the move at P15 is really
> #    best since the 3-3 point is still open. 
75a85,91
> #CATEGORY=BAD_PROBLEM
> #DESCRIPTION=Problem not clear enough.
> #SEVERITY=0
> #From STRATEGY:
> #     It is unclear that the recommended E9 is the best move. 
> #     W cannot cut at E11, or even threaten to cut and keep sente. 
> #     GNU Go 2.7.205 plays at G10 which is a good move.
86a103,111
> 
> #CATEGORY=PATTERN_TUNING
> #DESCRIPTION=N11 may not be biggest, but should be found.
> #SEVERITY=2
> #    2.7.179: This can surely be fixed with the addition of a
> #    pattern. After N11, the strings at M12 and N10 cannot both be 
> #    saved. This is not necessarily the biggest move on the board but 
> #    at least this move should be found.
> #
91a117,121
> 
> #CATEGORY=DYNAMIC_READING
> #DESCRIPTION=Atari is certainly better.
> #SEVERITY=4
> #PREDICESSOR=reading.tst#124
102a133,137
> 
> #CATEGORY=OWL_TUNING
> #DESCRIPTION=S13 is terrible - negative value.
> #SEVERITY=8
> #    2.7.179: Owl code finds Q11 but it remains undervalued at 17 points.
139a175,185
> 
> #CATEGORY=PATTERN_TUNING
> #DESCRIPTION=Blocking in the corner is better.
> #SEVERITY=3
> #    This test passed before 2.7.183, but for the wrong reason: 
> #    the move at C3 was allegedly an owl attack/defense point for 
> #    the dragon at D3, which of course it is not.
> #
> #    Thus this example should be regarded as having been already
> #    broken.
> #
193a240,247
> 
> #CATEGORY=STRATEGY
> #DESCRIPTION=L14 is not a terrible way to attack O13 group.
> #SEVERITY=2
> #Consider retiring this problem, or adding L14.  L14 is not terrible.
> #M11 is better than L4
> #    2.7.231: Broke again. M11 is overrated at 28 points. The
> #    large followup value is responsible.
197a252,265
> 
> #CATEGORY=STRATEGY
> #DESCRIPTION=N16 is better than M11.  Could be a tough attack, though.
> #SEVERITY=4
> #    2.7.179: After fixing strategy 40, the spurious defenses of D19
> #    will be removed. Now N16 will be the most valuable move. So 
> #    this should be fixed automatically when 40 is fixed.
> #
> #    2.7.180: this test passes.
> #
> #    2.7.220: N12 is found as an owl attack point for the S14 dragon.
> #    This attack will probably fail. More seriously, M16 is preferred
> #    to N16.
> #
210a279,295
> #CATEGORY=STRATEGY
> #DESCRIPTION=L4 is awkward.
> #SEVERITY=2
> #    2.7.179: Whether or not the recommended move at G2 is found, the 
> #    move at A6 is egregiously bad. In this situation an owl critical 
> #    dragon adjoins an owl dead one. We need to revisit our policy for 
> #    such situations. The reading code thinks the worm at B4 can be 
> #    defended, so this is an owl lunch.
> #
> #    2.7.180: After revision of semeai.c, the move at A6 is no longer
> #    found. Now GNU plays at M16 on the top. Arguably stabilizing
> #    the bottom at G2 or (worse but still OK) at J2 is better but
> #    I would class this result as ACCEPTABLE.
> #
> #    3.1.9: Best move here is tough.  GnuGo correctly tries
> #    to stabalize G3 group.
> #
227a313,315
> #CATEGORY=FUSEKI_STRATEGY
> #DESCRIPTION=B17 better than D17
> #SEVERITY=3
236a325,328
> 
> #CATEGORY=OWL_TUNING
> #DESCRIPTION=P9 is pointless compared to Q9.
> #SEVERITY=8
238a331
> # So much better, that I removed Q11 option -trevor
241c334
< #? [Q9|Q11]*
---
> #? [Q9]*
Index: STRATEGY
===================================================================
RCS file: /cvsroot/gnugo/gnugo/regression/STRATEGY,v
retrieving revision 1.3
diff -r1.3 STRATEGY
0a1,8
> THIS FILE CAN BE CONSIDERRED OBSOLETE
> 
> 3.1.9:  Moved all comments from here to strategy.tst
> 
> Comments on strategy2.tst still pending.
> 
> 
>