Re: [Bug-gnubg] Snowie error rates versus gnubg error rates

[Joern Thyssen]

On Mon, Apr 07, 2003 at 08:50:22PM -0400, Douglas Zare wrote
> Quoting Joern Thyssen <address@hidden>:
> > > 
> > > Otherwise, some people rated as "world-class" by bots will be rated
> > > "pigeon" by substantially stronger human players. I just analyzed a
> > > money session in which a player with an error rate under 3.0 (by
> > > Snowie rollouts) was estimated to be the favorite by about 0.12 ppg
> > > against a player with an error rate of 4.4. 
> > 
> > 0.12ppg seems like a lot when there is only 1.4 millipoints in
> > difference in the error rates. Were the games very long? On a 1-cube you
> > would need an average of 80 decisions per player to reach 0.12 ppg, so I
> > assume one of the players made a lot of errors on high cubes?
> 
> First, most of the initial doubles were taken (as is normal). There
> were 9 passes and 29 takes. It looks like the stronger player was
> intentionally doubling the weaker player in slightly early: the
> stronger player cashed twice and was doubled out 7 times. (Then again,
> some of the weaker player's takes should have been passes.)
> Nevertheless, the average cube level in the sense of the absolute
> error rate difference divided by the EMG error rate difference was
> less than 2.
> 
> Second, there were about 40 moves per game by Snowie's count, so 3.0
> millipoints per move absolute.
> 
> Third, I said the stronger player's error rate was under 3.0, not
> equal to 3.0.  It was actually lower. 

OK, that makes sense.

It's quite an interesting observation that a small difference of 1.4
millipoint per move actually make up a fairly good money game advantage.

For money game it may be an idea to use the unnormalised error rate (or
equivalently the ppg-advantage) to assign the "world class", "expert"
etc. labels. I'm just not sure how to generalise that to match play.

> You are free to set gnu's levels so that weak players are called
> "world-class," 

The new threshold are actually lower than the former ones, so we're
going the right way :-)


> but the elite backgammon players (at least on a good day) are much
> stronger than even what Snowie calls borderline WC. See, for example,
> the incomplete match between Malcolm Davis and Bob Zavoral currently
> on the front page of GammonVillage, with error rates of 3.007 and
> 3.337, despite the sloppy play at the end. Here are the Snowie error
> rates of other recent matches there:

3.5 seems like a reasonable threshold. That translates to approximately
5.0 on the gnubg scale.


Jørn