Re: [Bug-gnubg] Bug in sigmoid?

[Olivier Baur]

Le vendredi, 18 avr 2003, à 23:01 Europe/Paris, Joseph Heled a écrit :

> I am not sure about that. I have approximated exp(x) with 1+x, which 
> is mathematically sound and has an o(x^2) error, which is nice in the 
> [01] range the approximation is used. I think this is not the case for 
> 1/(1+x)

Yes, the approximation used for exp(x) in the current sigmoid function 
is correct and quite clever, since we have exp(x) = exp(X + dx) = 
exp(X).exp(dx) =  exp(X).(1 + dx) + o(dx^2), which requires only one 
exp(X) lookup.

Then, I don't understand what you mean about 1/(1+x)...

> Second, we need to re-train (we can't keep 2 nets around for practical 
> reasons).

Can't agree less :-)
And, apart from not being myself able to train a new net :-) , that's 
the reason why I used the current sigmoid function approximation to 
build the sigmoid2 lookup table, so we get the same results.

> Third, I need to see numbers on how faster this is on non scalar, 
> regular x86 machine.

I don't know if this is a typo, but please note a "scalar" processor is 
what you call a "regular" processor; on the other hand, "non-scalar" 
and "super-scalar" refer to vector-computing.

Please note the speed increase I have measured in sigmoid2 (+60%) was 
for a regular scalar (ie *non* vector) implementation (on a PPC G4 
processor); with a vector implementation of sigmoid2 (on the same 
processor), I actually got a whopping +250% speed increase...

So let me know what figures you get on a scalar x86 :-)


-- Olivier