[Octave-bug-tracker] [bug #39636] nbinpdf has floating point error

[Jordi Gutiérrez Hermoso]

Follow-up Comment #5, bug #39636 (project octave):

We obviously can't just compute the binomial coefficient using the gamma
function for all arguments, since it will fail for everything but the tiniest
numbers. If you look at the actual implementation, you'll see that we're using
 bincoeff to compute this, which in turn computes the binomial coefficients
using the usual log-gamma approach.

The bug, if there is one, is in the bincoeff function then. And again, the
problem is that we are essentially computing the value of the beta function
around a removable singularity, using sums of log-gamma functions. You'll note
that on line 92 here,

   
http://hg.savannah.gnu.org/hgweb/octave/file/26589abbc78d/scripts/miscellaneous/bincoeff.m#l91

since we've determined that n is not an integer, then we can take the lgamma
of it. But then we're actually very close to a singularity of the gamma
function. The whole thing still sort of manages to cancel and not give a
completely ridiculous answer, but I don't know if there is any sort of code
that we can implement here to make it better.

Do you have a suggestion?


    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?39636>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/