Re: [gnugo-devel] Regression test suites

[Gunnar Farnebäck]

John Washburn wrote:
> I would be interested in the regression test suite
> improvement/expansion.  I have done software testing for 11 years and
> have the CSQE certification from ASQ.

That would be welcome. I assume this is in response to this TODO item:

 * Extend the regression test suites.
   See the texinfo manual in the doc directory for a description of
   how to do this. In particular it would be useful with test suites
   for common life and death problems. Currently second line groups, L
   groups and the tripod shape are reasonably well covered, but there
   is for example almost nothing on comb formations, carpenter's
   square, and so on. Other areas where test suites would be most
   welcome are fuseki, tesuji, and endgame.

In addition to this it should be noted that the test suite is in need
of revision. The problem is that there are now very many tests and
some of those tend to generate lots of "regression noise", meaning
that any sufficiently disruptive patch will get a huge breakage where
a significant part is of the accidental type. In a few cases this is
because the tests are simply incorrect but the most common problem is
with anything-but-this-move (using the "!" negation in the correct
answer) tests, where since the test was entered the engine has started
to generate a different bad move. Then a good change that solves the
original problem may be undetected and something which solves the new
problem may cause a fail. The solution would in most cases be to
convert these negative answer tests into restricted_genmove tests
which only test for a specific problem.

> As for the super KO task on the tactical list, would using
> (hashes/message digests) to encode the board position of a given
> move be adequate.

I would rather characterize it as overkill. There is no reason to use
cryptographic hashes since Zobrist hashing is good enough, orders
of magnitude simpler, and already implemented.

With respect to your calculations my only comment is that you
shouldn't bother with probabilities less than 10^(-10) or so in this
context. They won't happen in your lifetime and even if they do there
will be no dire consequences. Chances are good that you won't even
notice.

Also, please post to this list in plain text, not html.

/Gunnar