Re: [Bug-gnubg] Is gnubg's game-analyisis function always reliable?

[Joseph Heled]

Hi Adi,

You are asking loads of good questions, most of them without definite
answer at this stage. It is hard to have an opinion with seeing the
actual numbers and the actual games, but here are my 2 cents. It is
possible your opponent(s) is playing a style which is harder for GNU
to analyze - which would result in him winning more and evaluated
lower. However so far I have seen only one consistent area where GNUbg
is totally lost - in deep backgames. Is that the case here?

Apart from that we humans are not good at perceiving the true nature
of randomness, especially in cases where the variance is high. I have
performed long simulation between two GNUbg versions, and sometimes
the "weaker" one would take a lead which would revert to the true
state only after many thousands of games, sometimes even longer. So 69
games is a very small number to be concerned about. My guess would be
that you will prevail if you play enough games against this opponent
if nothing changes.

Even GNUbg (0-ply) rank was oscillating between 1800 and 2200 when it
played on FIBS.

-Joseph

On Sun, Apr 26, 2009 at 4:23 AM, Adi Kadmon <address@hidden> wrote:
> Dear Developers
>
>
>
> I have Version GNU Backgammon 0.90-mingw 20090403, and I’ve been training a
> lot with it and finding it very strong.
>
>
>
> However, lately I’ve been encountering a phenomenon for which I cannot
> account. Analyzing the money-games (a single game each time) with some of my
> opponents using the feature “Analysis” (set to “supremo” 2-ply for checker
> play and to “grandmaster” 3-ply for cube decisions), I found consistently
> for some of the opponents that I was significantly better than they were by
> the analysis` measures, and nevertheless I consistently lost to them more
> than gain, sometimes in a ratio approaching 2:1 or 3:2!
>
>
>
> For instance, Analyzing 69(!) games with a certain opponent, with the above
> settings for the analysis function, I found the following:
>
>
>
> In 41 games I had an error rate mEMG per move (total: checker play + cube
> decisions) lower than his. The advantage was his only in 27, and in 1 game
> we came out equals.
>
>
>
> In 43 games I had an error rate mEMG per move (checker play) lower than his,
> whereas he had the advantage in 26 games only.
>
>
>
> In 29 games I had an error rate mEMG per cube decision lower than his,
> whereas he had the advantage in 21 only, and in 19 games we came out equals.
>
>
>
> Seeing all this, it was all the more disconcerting to find that on average I
> lost 0.2826 points per game during these 69 games!
>
>
>
> Note: the “marks” (“world-class…… awful”) for me and for him were consistent
> with the above analysis measures. The bot found my opponent to be luckier
> than me on average (unfortunately I didn’t keep the exact numbers of games
> on his side or mine) – but, for heaven’s sake, such luck through 69 game!!!
> With such an ostensible advantage on my side? It’s highly improbable. And
> this opponent is, as I said, not the only example.
>
>
>
> I’ve been trying to imagine an explanation for these facts, such as:
>
>
>
> (1)   That opponents, and a few more, happened to be definitely stronger
> than gnubg. But stronger than “supremo” and “grandmaster” in such a decided
> and consistent fashion – is it conceivable?
>
> (2)   Those opponents somehow cheated heavily on the dice (on Play65 site) –
> but I deem it very unlikely, and I’m not used to being paranoid at all…
>
> (3)   My opponents’ style was different from gnubg’s, to which I’m much more
> accustomed (and from which I indeed learned much), even though theoretically
> their moves are a bit inferior by its standards – and one has to “know how
> to win against their style” in order to succeed against them. But is such a
> thing likely? I would have rather thought that if by a very strong bot’s
> standards my moves are superior on average, it’s not a relativistic but a
> definite superiority and should beat a “different style”.
>
> (4)   My errors occur in such early (or late) stages in the game in
> comparison with the opponent’s errors, that my errors are “more costly”,
> more decisive for the final result. However, I doubt this too: Wouldn’t the
> bot than have found my mistakes statistically very grave for the relevant
> positions, thereby affecting the above measures accordingly so that on
> average they again should turn out inferior, not superior?!
>
>
>
>
>
> Well, I trust your vast knowledge and experience would yield a suitable
> explanation, or at least point to some possible explanations.
>
>
>
> Please do help!
>
>
>
> Sincerely,
>
>
>
> Adi
>
>
>
> _______________________________________________
> Bug-gnubg mailing list
> address@hidden
> http://lists.gnu.org/mailman/listinfo/bug-gnubg
>
>