[gnugo-devel] arend_1_16.3: New regression test suite

[Arend Bayer]

I have been analysing somewhat systematically the blunders by GNU Go
in Stefan Mertin's private 13x13 tournament (games agains GoIntellect 10.0,
Katsunari, Goliath 3.5). To qoute my own comment :-) in the .tst-file:
# I think they should be
# quite useful as tests. 13x13-positions are specific as problems, and
# they are quickly evaluated both by GNU Go and by us when maintaining.
# (And of course, these are games against some opponents we would like
# to beat :-) )
# I also added a few test cases where GNU Go played right in the game,
# but went wrong in a replay by a newer version (3.14 / 3.15).

- Arend

For the record, 3.1.15 has a gain of 28 improvements vs 21 new failures
compared to 3.0.0. Those doing owl-tuning might want to look at 86 and 89,
the two new OWL-failures (haven't checked the reason for the failure
yet).
(Expected results are set according to GNU Go 3.1.15.)

3.0.0 breaks on 13x13 as follows:
2 unexpected FAIL: Correct 'E9|E10', got 'K5'
3 unexpected FAIL: Correct '!E2', got 'E2'
5 unexpected FAIL: Correct 'N10|N9|M1', got 'K3'
6 unexpected PASS!
14 unexpected PASS!
15 unexpected PASS!
16 unexpected FAIL: Correct 'C6|C5', got 'F10'
17 unexpected PASS!
19 unexpected FAIL: Correct '!N3', got 'N3'
22 unexpected FAIL: Correct 'C9', got 'B9'
25 unexpected FAIL: Correct 'J5', got 'L2'
26 unexpected FAIL: Correct 'J5', got 'L4'
28 unexpected FAIL: Correct 'L10', got 'K12'
29 unexpected FAIL: Correct 'C8', got 'C9'
30 unexpected FAIL: Correct '!C11', got 'C11'
34 unexpected PASS!
35 unexpected PASS!
36 unexpected FAIL: Correct 'C5', got 'B13'
38 unexpected FAIL: Correct 'G11', got 'E4'
40 unexpected PASS!
41 unexpected FAIL: Correct '!C8', got 'C8'
42 unexpected FAIL: Correct 'N3|N2|L4|L3|L2|L1|M1|N1|M3', got 'C7'
47 unexpected FAIL: Correct '!G11', got 'G11'
48 unexpected FAIL: Correct 'K7', got 'K6'
50 unexpected PASS!
51 unexpected FAIL: Correct 'L5', got 'J10'
52 unexpected PASS!
53 unexpected FAIL: Correct 'C6', got 'B6'
57 unexpected PASS!
58 unexpected PASS!
59 unexpected PASS!
61 unexpected FAIL: Correct 'K8', got 'M12'
63 unexpected PASS!
65 unexpected PASS!
67 unexpected PASS!
68 unexpected FAIL: Correct 'A8', got 'C9'
71 unexpected FAIL: Correct 'J11', got 'L12'
72 unexpected FAIL: Correct 'J10', got 'G9'
73 unexpected PASS!
75 unexpected FAIL: Correct '!D11', got 'D11'
76 unexpected FAIL: Correct 'K6|L6|J3', got 'C12'
77 unexpected PASS!
80 unexpected PASS!
81 unexpected FAIL: Correct 'M8', got 'K1'
82 unexpected PASS!
85 unexpected FAIL: Correct 'D2', got 'K7'
88 unexpected PASS!
89 unexpected PASS!
90 unexpected FAIL: Correct 'A5', got 'E5'


- regression tests added from games at Stefan Mertin's 13x13 tournament

Index: regression/Makefile.am
===================================================================
RCS file: /cvsroot/gnugo/gnugo/regression/Makefile.am,v
retrieving revision 1.19
diff -u -r1.19 Makefile.am
--- regression/Makefile.am      2001/11/28 18:35:18     1.19
+++ regression/Makefile.am      2001/12/07 23:52:01
@@ -8,7 +8,7 @@
       niki.tst trevor.tst tactics.tst buzco.tst \
       capture.tst connect.tst global.tst vie.tst \
       owl_rot.tst score_rot.tst strategy_rot.tst connect_rot.tst arend.tst \
-      trevora.tst semeai.tst
+      trevora.tst semeai.tst 13x13.tst
 
 noinst_SCRIPTS = eval.sh regress.sh test.sh
 
@@ -183,6 +183,9 @@
 semeai: semeai.tst
        $(srcdir)/eval.sh semeai.tst
 
+13x13: 13x13.tst
+       $(srcdir)/eval.sh 13x13.tst
+
 all_batches: first_batch second_batch third_batch fourth_batch
 
 first_batch: reading.tst owl.tst owl_rot.tst ld_owl.tst optics.tst filllib.tst 
atari_atari.tst connection.tst blunder.tst strategy.tst
@@ -229,5 +232,6 @@
        $(srcdir)/regress.sh $(srcdir) global.tst
        $(srcdir)/regress.sh $(srcdir) vie.tst
        $(srcdir)/regress.sh $(srcdir) arend.tst
+       $(srcdir)/regress.sh $(srcdir) 13x13.tst
        $(srcdir)/regress.sh $(srcdir) trevora.tst
        $(srcdir)/regress.sh $(srcdir) strategy4.tst
Index: regression/Makefile.in
===================================================================
RCS file: /cvsroot/gnugo/gnugo/regression/Makefile.in,v
retrieving revision 1.32
diff -u -r1.32 Makefile.in
--- regression/Makefile.in      2001/11/30 14:42:02     1.32
+++ regression/Makefile.in      2001/12/07 23:52:02
@@ -75,7 +75,7 @@
 glibconfig = @glibconfig@
 perl = @perl@
 
-TST = life.tst owl.tst optics.tst reading.tst strategy.tst       ld_owl.tst 
connection.tst        neurogo.tst arb.tst strategy2.tst strategy3.tst 
rosebud.tst       heikki.tst golife.tst dniwog.tst strategy4.tst       
nicklas1.tst nicklas2.tst nicklas3.tst nicklas4.tst nicklas5.tst       
filllib.tst arion.tst endgame.tst viking.tst ego.tst atari_atari.tst       
score.tst manyfaces.tst cutstone.tst newscore.tst blunder.tst       niki.tst 
trevor.tst tactics.tst buzco.tst       capture.tst connect.tst global.tst 
vie.tst       owl_rot.tst score_rot.tst strategy_rot.tst connect_rot.tst 
arend.tst       trevora.tst semeai.tst
+TST = life.tst owl.tst optics.tst reading.tst strategy.tst       ld_owl.tst 
connection.tst        neurogo.tst arb.tst strategy2.tst strategy3.tst 
rosebud.tst       heikki.tst golife.tst dniwog.tst strategy4.tst       
nicklas1.tst nicklas2.tst nicklas3.tst nicklas4.tst nicklas5.tst       
filllib.tst arion.tst endgame.tst viking.tst ego.tst atari_atari.tst       
score.tst manyfaces.tst cutstone.tst newscore.tst blunder.tst       niki.tst 
trevor.tst tactics.tst buzco.tst       capture.tst connect.tst global.tst 
vie.tst       owl_rot.tst score_rot.tst strategy_rot.tst connect_rot.tst 
arend.tst       trevora.tst semeai.tst 13x13.tst
 
 
 noinst_SCRIPTS = eval.sh regress.sh test.sh
@@ -356,6 +356,9 @@
 semeai: semeai.tst
        $(srcdir)/eval.sh semeai.tst
 
+13x13: 13x13.tst
+       $(srcdir)/eval.sh 13x13.tst
+
 all_batches: first_batch second_batch third_batch fourth_batch
 
 first_batch: reading.tst owl.tst owl_rot.tst ld_owl.tst optics.tst filllib.tst 
atari_atari.tst connection.tst blunder.tst strategy.tst
@@ -402,6 +405,7 @@
        $(srcdir)/regress.sh $(srcdir) global.tst
        $(srcdir)/regress.sh $(srcdir) vie.tst
        $(srcdir)/regress.sh $(srcdir) arend.tst
+       $(srcdir)/regress.sh $(srcdir) 13x13.tst
        $(srcdir)/regress.sh $(srcdir) trevora.tst
        $(srcdir)/regress.sh $(srcdir) strategy4.tst

arend_1_16.3.tar.gz
Description: New files