Re: [Bug-gnubg] Possible error in SSE sigmoid calculation

[Philippe Michel]

On Wed, 21 Dec 2011, Joseph Heled wrote:

> I find it hard to believe.
>
> There is a simple test - if the two implementations give the same vale for
> the same position, ply 0.

Analyzing and exporting to text a sample match at 0-ply, I get this as a 
rough measure of number of discrepancies between the following options :

nosse : scalar code without SSE intrinsics
ssecvt : SSE intrinsics with cvtps instruction
ssecvtt : SSE intrinsics with cvttps intruction

% diff nosse.txt ssecvt.txt | wc
   16360   94346  681212
% diff nosse.txt ssecvtt.txt | wc
    2434   12056   90952

The differences between nosse and ssecvtt are due, I think, to the usage 
of a reciprocal instead of a true division in 1/(1+exp(x)). I suppose this 
was found acceptable at the time and would dissappear if everything was 
compiled with the -ffast-math flag. Maybe one could use something like

#ifdef __FAST_MATH__
... use reciprocal intrinsic
#else
... use divide intrinsic
#endif

in neuralnetsse.c

The discrepancies in ssecvt, on the other hand, are not only about seven 
times more common but come from the flawed approximation of exp(x) in 
addition to the above shortcut.

Looking cursorily at the exports, I found only occasionnal 0.001 
differencies in the ssecvtt case, without moves swapped.

In the ssecvt case, there were some almost-equal moves ordered differently 
(in the exported top five choices). I didn't found cases where the top 
move was changed or equity differences of more than 0.001, but I'm 
convinced the former, at least, could happen.

These are still pretty small differences. In the final summary of these 
analyses for instance I get :

nosse :   Error total EMG (MWC)  -4.142 (-14.676%) -4.845 (-15.479%)
ssecvt :  Error total EMG (MWC)  -4.142 (-14.682%) -4.828 (-15.344%)
ssecvtt : Error total EMG (MWC)  -4.143 (-14.683%) -4.846 (-15.484%)