Re: [gnubg] Help with a new MET (2)

[Joseph Heled]

On Wed, 27 Nov 2019 at 13:25, Ian Dunstan <address@hidden> wrote:
>
> Hi Joseph,
>
> "
> That is why I, personally, would like to see proof it really matters. like:
>
>   -- Take a weak player - with the "weakest" MET file and the
> "strongest" - and show a significant difference.
>
> "
>
> I understand your viewpoint. When Gergely and I can get the scripts and any 
> bug-fixes sorted we will first do some preliminary testing as you suggest.
>
> I think 0-ply v 0-ply using the Woolsey MET vs the Kaz-XG2 MET might be the 
> way to start.
>
> Then 2-ply v 2-ply using the Woolsey MET vs the Kaz-XG2 MET is probably 
> needed as well for comparison.
>
> Let's see what comes from these sorts of tests and move on from there.
>
> I think I probably need to ask Xavier how he arrived at his supposedly 
> 'significant difference' of 0.4 ELO between the ExtremGammon MET and the 
> Roc-Kaz MET.
>

GNUBG ply-0 might not be weak enough :) I might be wrong here, but I
"wanted" a weak player in the belief it will be easier to establish
statistical significance. But I might be wrong and it is the other
way. To get a weaker player you probably need to use a "weaker/lesser
trained net".

-Joseph

> Cheers, Ian.
> On Wednesday, 27 November 2019, 05:59:09 am AEDT, Joseph Heled 
> <address@hidden> wrote:
>
>
> Hi Ian and bg fans,
>
> I would like to clarify my own view RE your post and the other replies.
>
> Any BG playing bot is a complex entity. GNUBG uses two-sided bearoff
> tables, one-sided bearoff tables, three different types of neural
> nets, not to mention my "pruning nets" --  And all this just to guess
> the cubeless probabilities. For match play it uses the MET file and
> *an algorithm* to covert cubeless to cubeful. GNUBG uses the
> code+formulas by Joern Thyssen, which I never understood fully, and
> was always slightly skeptical of. My own cube code is based on linear
> interpolating, based on the ideas here
> (https://bkgm.com/articles/met.html) which I think is the right
> approach (danalyze.cc in gnubg-nn).
>
> Yes, those are details, but they matter. The playing abilities of a
> bot are a complex emergent property of the system. The MET table is
> just one part, and focusing on it is <very-personal-opinion>enormously
> premature</very-personal-opinion>. My own experience indicates the
> system as a whole is not that sensitive to the MET. That is why I,
> personally, would like to see proof it really matters. like:
>
>   -- Take a weak player - with the "weakest" MET file and the
> "strongest" - and show a significant difference.
>
> Then we can start debating if it is worth while to "improve" the MET.
> We have to use BOTS to develop this even if it is intended for human
> use. Humans can't generate enough test data.
>
> I have more thoughts, but that is enough on this topic :)
>
> -Joseph
>
> On Tue, 26 Nov 2019 at 22:44, Ian Dunstan <address@hidden> wrote:
> >
> > Hi team,
> >
> > I have tried several subscription attempts to the email list directly, I 
> > think, without success. After finding the archives I see that there have 
> > been a number of replies to my first post. I didn't read them until very 
> > recently I wasn't ignoring your feedback, I just had not viewed it. Thank 
> > you, all, for what I received. Also, I apologise that this post almost 
> > certainly starts a new thread, though I considered it more important to 
> > make a response now.
> >