Re: [Bug-gnubg] Huge evaluation difference

[Philippe Michel]

On Mon, Mar 18, 2019 at 10:25:52AM +0100, Terje Pedersen wrote:
 
> I just ran into a huge evaluation difference between XG and Gnu BG:
> 
> XGID=-B----CFC----A-------acbb-:1:1:1:16:0:0:0:5:10
> 
> I got dinged with a -0.684 error here while XG says it is a -0.0013
> error to play 13/6.
> 
> Desktop version says it is:
> 
>     7. Cubeful 0-ply    13/6                         Eq.:  -2,264 ( -0,328)
>        0,000 0,000 0,000 - 1,000 0,964 0,000
>         0-ply cubeful prune [expert]
> 
> but still wildly off. Perhaps difficult to fix these kind of problems?

The position is of a kind that was probably not common in the racing 
neural net training database. X's formation just doesn't happen in real 
play, does it ?

 GNU Backgammon  Position ID: 2wUAAAb3OwgAAA
                 Match ID   : UQmnAAAAAAAE
 +13-14-15-16-17-18------19-20-21-22-23-24-+     O: GNUbg
 | X                |   |       O  O  O  O | OO  0 points
 |                  |   |          O  O  O | OO  
 |                  |   |          O       | O   
 |                  |   |                  | O   
 |                  |   |                  | O  
v|                  |BAR|                  |     5 point match
 |                6 |   |                  |    
 |                X |   |                  |     
 |             X  X |   | X                |     
 |             X  X |   | X              X |     Rolled 61
 |             X  X |   | X              X |     0 points
 +12-11-10--9--8--7-------6--5--4--3--2--1-+     X: You (Cube: 2)
                    Pip counts : O 19, X 99

"Fixing" this at the neural net level doesn't look easy. Adding a few 
hundreds or thousands of similar positions in the training data would 
probably help but it is difficult to guess in advance if the improvement 
would be significant.

It wouldn't change the program's strength (it doesn't get into these 
positions) but may avoid or at least mitigate this kind of embarrassing 
result when analysing.

What you can do if it is important for you (for the analysis feature of 
Backgamon Studio, I suppose) is to use the largest one-sided bearoff 
database, available there : 
ftp://ftp.demon.nl/pub/Museum/Demon/games/gnubg/databases/os/

If the download size is an issue, you can compute it yourself with the
makebearoff utility from gnubg ; os13.bd (with the back checker up to 
the midpoint, as in your position) should take about one day, smaller 
ones would be much faster.

With it, I get :

    1. Cubeful 2-ply    13/6                         Eq.: -2.201
       0.000 0.000 0.000 - 1.000 0.913 0.000
        2-ply cubeful prune [world class]
    2. Cubeful 2-ply    8/2 7/6                      Eq.: -2.204 (-0.003)
       0.000 0.000 0.000 - 1.000 0.915 0.000
        2-ply cubeful prune [world class]
    3. Cubeful 2-ply    7/6 7/1                      Eq.: -2.213 (-0.013)
       0.000 0.000 0.000 - 1.000 0.923 0.000
        2-ply cubeful prune [world class]
    4. Cubeful 2-ply    13/12 8/2                    Eq.: -2.234 (-0.034)
       0.000 0.000 0.000 - 1.000 0.939 0.000
        2-ply cubeful prune [world class]
    5. Cubeful 2-ply    13/12 7/1                    Eq.: -2.240 (-0.039)
       0.000 0.000 0.000 - 1.000 0.943 0.000
        2-ply cubeful prune [world class]
    6. Cubeful 2-ply    13/7 6/5                     Eq.: -2.294 (-0.093)
       0.000 0.000 0.000 - 1.000 0.984 0.000
        2-ply cubeful prune [world class]

It may seem radical to use a 1.5 Gb database just for this but, besides 
the disk space needed, it is essentially free. The file is not read, it 
is merely mapped into memory and the relevant blocks are read only when 
needed.