Re: precision problem with mex file

[CdeMills]

fredounet wrote
> A=351.40999999999985448084771633148193359375000000000000*351.40999999999985448084771633148193359375000000000000;
> A=A+105.45999999999912688508629798889160156250000000000000*105.45999999999912688508629798889160156250000000000000;

This does not make sense. Octave uses IEEE-754 double precision numbers,
which means 53 bits of mantissa, and around 16 decimal digits. Your numbers
seems to have 40 significant digits, this is not supported at the present
time. "Things are working" means that you can not detect truncation effects.
Do not forget that numbers like 351.41 are NOT exact, finite-length in
binary representation. I mean that you require an infinite sequence of
digits to approximate it in base ten; 0.410156250 is an exact number in base
two.

If you really want to work at that level of accuracy, I'm porting a package
called mptoolbox; have a look at hg.code.sf.net/p/octave/multi-precision for
a sort-of working version.

Regards

Pascal



--
View this message in context: 
http://octave.1599824.n4.nabble.com/precision-problem-with-mex-file-tp4655798p4655801.html
Sent from the Octave - General mailing list archive at Nabble.com.