Re: [Bug-gnubg] pubeval benchmark

[Joseph Heled]

Speaking of bot to bot comparisons...

Anyone knows anything about the "raw" strength of the XG variants? By
raw I mean a the 0-ply equivalent, i.e. without any lookup.

I guess what I am asking is: how much of  XG strength comes from
quality of implementation, and how much from the basic net evaluation?

-Joseph

On 20 January 2012 12:03, Nikos Papahristou <address@hidden> wrote:
> Interesting.
> For comparison reasons with Palamedes and TD-Gammon, backgammons should
> count as gammons. This changes the second performance metric to  0.6046ppg.
>
> Nikos
>
> On 20/1/2012 0:20 πμ, Joseph Heled wrote:
>> Thanks for all of you who chipped in!
>>
>> Playing a 1 point match at 0 ply, gnubg won 7193 of 10000 matches,
>> which is 0.4386 ppg in my book.
>> Playing 10000 money sessions (gammons count, no cube) gnubg got 0.6296
>> ppg (the breakdown is 4504, 2298, 285   2202, 676, 35 )
>>
>> -Joseph
>>
>> On 20 January 2012 06:16, Nikos Papahristou <address@hidden> wrote:
>>> My evaluations of a starting roll 3-1 are exactly the same with Mark.
>>> Joseph, my implementation of pubeval doesn't agree with the moves
>>> selection in the games you sent.
>>> In the first game, the first roll of pubeval (4-5) is played as 24/15,
>>> and in the second game the first roll (52) is played as 13/8, 13/11.
>>>
>>> Nikos
>>>
>>> On 19/1/2012 17:14 μμ, Mark Higgins wrote:
>>>> I get something different for a 3-1 off the starting board. Sorry for
>>>> the non-standard format but hopefully it's intelligible. I'm printing
>>>> out the layout and the pubeval value, ordered by descending value
>>>> (based on my pubeval implementation). 8-5 6-5 is the best move.
>>>>
>>>> *|| | | | | ||| | | | | ||*
>>>> *|o *| * o|*
>>>> *|o *| * o|*
>>>> *| *| * o|*
>>>> *| *| o|*
>>>> *| *| o|*
>>>> *| | |*
>>>> *| | |*
>>>> *| | |*
>>>> *| | *|*
>>>> *| o| *|*
>>>> *| o| *|*
>>>> *|* o o| o *|*
>>>> *|* o o| o *|*
>>>> *|| | | | | ||| | | | | ||*
>>>> *Value 10.3431*
>>>>
>>>> *|| | | | | ||| | | | | ||*
>>>> *|o o *| * o|*
>>>> *| *| * o|*
>>>> *| *| * o|*
>>>> *| *| o|*
>>>> *| *| o|*
>>>> *| | |*
>>>> *| | |*
>>>> *| | |*
>>>> *| o| *|*
>>>> *| o| *|*
>>>> *| o| o *|*
>>>> *|* o| o *|*
>>>> *|* o| o *|*
>>>> *|| | | | | ||| | | | | ||*
>>>> *Value 7.22287*
>>>>
>>>> *|| | | | | ||| | | | | ||*
>>>> *|o *| * o|*
>>>> *|o *| * o|*
>>>> *| *| * o|*
>>>> *| *| o|*
>>>> *| *| |*
>>>> *| | |*
>>>> *| | |*
>>>> *| | |*
>>>> *| o| *|*
>>>> *| o| *|*
>>>> *| o| o *|*
>>>> *|* o| o *|*
>>>> *|* o| o o *|*
>>>> *|| | | | | ||| | | | | ||*
>>>> *Value 6.73507*
>>>>
>>>> *|| | | | | ||| | | | | ||*
>>>> *| o o *| * o|*
>>>> *| *| * o|*
>>>> *| *| * o|*
>>>> *| *| o|*
>>>> *| *| o|*
>>>> *| | |*
>>>> *| | |*
>>>> *| | |*
>>>> *| o| *|*
>>>> *| o| *|*
>>>> *| o| o *|*
>>>> *|* o| o *|*
>>>> *|* o| o *|*
>>>> *|| | | | | ||| | | | | ||*
>>>> *Value 6.64798*
>>>>
>>>> *|| | | | | ||| | | | | ||*
>>>> *|o o *| * o|*
>>>> *| *| * o|*
>>>> *| *| * o|*
>>>> *| *| o|*
>>>> *| *| |*
>>>> *| | |*
>>>> *| | |*
>>>> *| | |*
>>>> *| o| *|*
>>>> *| o| *|*
>>>> *| o| o *|*
>>>> *|* o| o *|*
>>>> *|* o| o o *|*
>>>> *|| | | | | ||| | | | | ||*
>>>> *Value 6.61369*
>>>>
>>>> *|| | | | | ||| | | | | ||*
>>>> *|o o *| * o|*
>>>> *| *| * o|*
>>>> *| *| * o|*
>>>> *| *| o|*
>>>> *| *| o|*
>>>> *| | |*
>>>> *| | |*
>>>> *| | |*
>>>> *| | *|*
>>>> *| o| *|*
>>>> *| o| o *|*
>>>> *|* o| o *|*
>>>> *|* o o| o *|*
>>>> *|| | | | | ||| | | | | ||*
>>>> *Value 6.44695*
>>>>
>>>> *|| | | | | ||| | | | | ||*
>>>> *|o o *| * o|*
>>>> *| *| * o|*
>>>> *| *| * o|*
>>>> *| *| o|*
>>>> *| *| o|*
>>>> *| | |*
>>>> *| | |*
>>>> *| | |*
>>>> *| o| *|*
>>>> *| o| *|*
>>>> *| o| *|*
>>>> *|* o| o *|*
>>>> *|* o|o o *|*
>>>> *|| | | | | ||| | | | | ||*
>>>> *Value 6.17616*
>>>>
>>>> *|| | | | | ||| | | | | ||*
>>>> *|o *| * o|*
>>>> *|o *| * o|*
>>>> *| *| * o|*
>>>> *| *| o|*
>>>> *| *| |*
>>>> *| | |*
>>>> *| | |*
>>>> *| | |*
>>>> *| | *|*
>>>> *| o| *|*
>>>> *| o| o *|*
>>>> *|* o| o *|*
>>>> *|* o o| o o *|*
>>>> *|| | | | | ||| | | | | ||*
>>>> *Value 6.09006*
>>>>
>>>> *|| | | | | ||| | | | | ||*
>>>> *|o *| * o|*
>>>> *|o *| * o|*
>>>> *| *| * o|*
>>>> *| *| o|*
>>>> *| *| |*
>>>> *| | |*
>>>> *| | |*
>>>> *| | |*
>>>> *| o| *|*
>>>> *| o| *|*
>>>> *| o| *|*
>>>> *|* o| o *|*
>>>> *|* o|o o o *|*
>>>> *|| | | | | ||| | | | | ||*
>>>> *Value 5.81926*
>>>>
>>>> *|| | | | | ||| | | | | ||*
>>>> *|o o *| * o|*
>>>> *| *| * o|*
>>>> *| *| * o|*
>>>> *| *| o|*
>>>> *| *| o|*
>>>> *| | |*
>>>> *| | |*
>>>> *| | |*
>>>> *| o| *|*
>>>> *| o| *|*
>>>> *| o| *|*
>>>> *|* o| o *|*
>>>> *|* o o| o *|*
>>>> *|| | | | | ||| | | | | ||*
>>>> *Value 5.76592*
>>>>
>>>> *|| | | | | ||| | | | | ||*
>>>> *|o *| * o|*
>>>> *|o *| * o|*
>>>> *| *| * o|*
>>>> *| *| o|*
>>>> *| *| o|*
>>>> *| | |*
>>>> *| | |*
>>>> *| | |*
>>>> *| | *|*
>>>> *| | *|*
>>>> *| o| o *|*
>>>> *|* o| o *|*
>>>> *|* o o o| o *|*
>>>> *|| | | | | ||| | | | | ||*
>>>> *Value 5.65459*
>>>>
>>>> *|| | | | | ||| | | | | ||*
>>>> *|o *| * o|*
>>>> *|o *| * o|*
>>>> *| *| * o|*
>>>> *| *| o|*
>>>> *| *| o|*
>>>> *| | |*
>>>> *| | |*
>>>> *| | |*
>>>> *| o| *|*
>>>> *| o| *|*
>>>> *| o| *|*
>>>> *|* o| o *|*
>>>> *|* o o| o *|*
>>>> *|| | | | | ||| | | | | ||*
>>>> *Value 5.63356*
>>>>
>>>> *|| | | | | ||| | | | | ||*
>>>> *|o o *| * o|*
>>>> *| *| * o|*
>>>> *| *| * o|*
>>>> *| *| o|*
>>>> *| *| o|*
>>>> *| | |*
>>>> *| | |*
>>>> *| | |*
>>>> *| | *|*
>>>> *| o| *|*
>>>> *| o| o *|*
>>>> *|* o| o *|*
>>>> *|* o o| o *|*
>>>> *|| | | | | ||| | | | | ||*
>>>> *Value 5.55805*
>>>>
>>>> *|| | | | | ||| | | | | ||*
>>>> *|o *| * o|*
>>>> *|o *| * o|*
>>>> *| *| * o|*
>>>> *| *| o|*
>>>> *| *| o|*
>>>> *| | |*
>>>> *| | |*
>>>> *| | |*
>>>> *| | *|*
>>>> *| o| *|*
>>>> *| o| o *|*
>>>> *|* o| o *|*
>>>> *|* o o| o *|*
>>>> *|| | | | | ||| | | | | ||*
>>>> *Value 5.44036*
>>>>
>>>> *|| | | | | ||| | | | | ||*
>>>> *|o *| * o|*
>>>> *|o *| * o|*
>>>> *| *| * o|*
>>>> *| *| o|*
>>>> *| *| o|*
>>>> *| | |*
>>>> *| | |*
>>>> *| | |*
>>>> *| | *|*
>>>> *| o| *|*
>>>> *| o| *|*
>>>> *|* o| o *|*
>>>> *|* o o|o o *|*
>>>> *|| | | | | ||| | | | | ||*
>>>> *Value 4.76362*
>>>>
>>>> *|| | | | | ||| | | | | ||*
>>>> *|o *| * o|*
>>>> *|o *| * o|*
>>>> *| *| * o|*
>>>> *| *| o|*
>>>> *| *| o|*
>>>> *| | |*
>>>> *| | |*
>>>> *| | |*
>>>> *| o| *|*
>>>> *| o| *|*
>>>> *| o| *|*
>>>> *|* o| *|*
>>>> *|* o o|o o *|*
>>>> *|| | | | | ||| | | | | ||*
>>>> *Value 3.78692*
>>>>
>>>>
>>>> On Jan 19, 2012, at 9:57 AM, Joseph Heled wrote:
>>>>
>>>>> Here are all the possibilities for a 3,1 from the initial position. I
>>>>> am worried that 8-5 6-5 sneaks at number 6
>>>>>
>>>>> score gnubg id position
>>>>> 8.00384996831 4HPwATDgc/ABIQ (0, 1, 0, 0, 0, 1, -5, 0, -3, 0, 0, 0, 5,
>>>>> -5, 0, 0, 0, 3, 0, 5, 0, 0, 0, 0, -2, 0)
>>>>> 7.83777993917 4HPwATDEc/ABKA (0, 1, 1, 0, 0, 0, -5, 0, -3, 0, 0, 0, 5,
>>>>> -5, 0, 0, 0, 3, 0, 4, 0, 0, 1, 0, -2, 0)
>>>>> 7.81063996255 4HPwATDgc/ABEg (0, 0, 1, 0, 1, 0, -5, 0, -3, 0, 0, 0, 5,
>>>>> -5, 0, 0, 0, 3, 0, 5, 0, 0, 0, 0, -2, 0)
>>>>> 7.75730997324 4HPwATDgc+IBKA (0, 1, 1, 0, 0, 0, -5, 0, -3, 0, 0, 0, 4,
>>>>> -5, 0, 1, 0, 3, 0, 5, 0, 0, 0, 0, -2, 0)
>>>>> 7.43968993425 4HPwATDQZ/ABKA (0, 1, 1, 0, 0, 0, -5, 0, -3, 0, 0, 0, 5,
>>>>> -5, 0, 0, 0, 2, 0, 5, 1, 0, 0, 0, -2, 0)
>>>>> 6.84054994583 4HPwATCwZ/ABMA (0, 2, 0, 0, 0, 0, -5, 0, -3, 0, 0, 0, 5,
>>>>> -5, 0, 0, 0, 2, 0, 4, 2, 0, 0, 0, -2, 0)
>>>>> 6.81688992679 4HPwATDga/ABIg (0, 1, 0, 0, 1, 0, -5, 0, -3, 0, 0, 0, 5,
>>>>> -5, 0, 0, 0, 2, 1, 5, 0, 0, 0, 0, -2, 0)
>>>>> 6.67967991531 4HPwATDQc/ABIg (0, 1, 0, 0, 1, 0, -5, 0, -3, 0, 0, 0, 5,
>>>>> -5, 0, 0, 0, 3, 0, 4, 1, 0, 0, 0, -2, 0)
>>>>> 6.67962995172 4HPwATDCc/ABMA (0, 2, 0, 0, 0, 0, -5, 0, -3, 0, 0, 0, 5,
>>>>> -5, 0, 0, 0, 3, 0, 4, 0, 0, 0, 1, -2, 0)
>>>>> 6.24834996462 4HPwATDIZ/ABMA (0, 2, 0, 0, 0, 0, -5, 0, -3, 0, 0, 0, 5,
>>>>> -5, 0, 0, 0, 2, 0, 5, 0, 1, 0, 0, -2, 0)
>>>>> 6.10885995626 4HPwATDgc+EBMA (0, 2, 0, 0, 0, 0, -5, 0, -3, 0, 0, 0, 4,
>>>>> -5, 0, 0, 1, 3, 0, 5, 0, 0, 0, 0, -2, 0)
>>>>> 4.3719098568 4HPwATDEa/ABMA (0, 2, 0, 0, 0, 0, -5, 0, -3, 0, 0, 0, 5,
>>>>> -5, 0, 0, 0, 2, 1, 4, 0, 0, 1, 0, -2, 0)
>>>>> 4.29143989086 4HPwATDga+IBMA (0, 2, 0, 0, 0, 0, -5, 0, -3, 0, 0, 0, 4,
>>>>> -5, 0, 1, 0, 2, 1, 5, 0, 0, 0, 0, -2, 0)
>>>>> 4.23469984531 4HPwATCkc/ABMA (0, 2, 0, 0, 0, 0, -5, 0, -3, 0, 0, 0, 5,
>>>>> -5, 0, 0, 0, 3, 0, 3, 1, 0, 1, 0, -2, 0)
>>>>> 4.15422987938 4HPwATDQc+IBMA (0, 2, 0, 0, 0, 0, -5, 0, -3, 0, 0, 0, 4,
>>>>> -5, 0, 1, 0, 3, 0, 4, 1, 0, 0, 0, -2, 0)
>>>>> 3.22565984726 4HPwATDQV/ABMA (0, 2, 0, 0, 0, 0, -5, 0, -3, 0, 0, 0, 5,
>>>>> -5, 0, 0, 0, 1, 1, 5, 1, 0, 0, 0, -2, 0)
>>>>>
>>>>>
>>>>> On 20 January 2012 03:45, Mark Higgins <address@hidden
>>>>> <mailto:address@hidden>> wrote:
>>>>>> :) I'll give it a try and check the couple games you sent.
>>>>>>
>>>>>>
>>>>>> On Jan 19, 2012, at 9:39 AM, Joseph Heled wrote:
>>>>>>
>>>>>>> I was asking for help, not for more work.
>>>>>>>
>>>>>>> For example, anyone who implemented play against pubeval can check a
>>>>>>> few of the moves I sent.
>>>>>>>
>>>>>>> -Joseph
>>>>>>>
>>>>>>> On 20 January 2012 03:35, Mark Higgins <address@hidden
>>>>>>> <mailto:address@hidden>> wrote:
>>>>>>>> I used the following post to define pubeval; is this the proper
>>>>>>>> definition still:
>>>>>>>>
>>>>>>>> http://www.bkgm.com/rgb/rgb.cgi?view+610
>>>>>>>>
>>>>>>>> ?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Jan 19, 2012, at 8:31 AM, Joseph Heled wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I just run 10000 1 point match games between gnubg and pubeval. I
>>>>>>>>> think this is the most fair, since there are no gammons of cubes
>>>>>>>>> involved.
>>>>>>>>>
>>>>>>>>> However, gnubg won 9840 of those, which makes me think something
>>>>>>>>> is wrong.
>>>>>>>>>
>>>>>>>>> Can someone help me verify that pubeval makes the "right" moves?
>>>>>>>>>
>>>>>>>>> I can send a few games in oldmoves format, or can check specific
>>>>>>>>> moves.
>>>>>>>>>
>>>>>>>>> I am attaching 2 games as examples.
>>>>>>>>>
>>>>>>>>> -Joseph
>>>>>>>>> <game1.fibs><game2.fibs>
>