RE: precision of floor(a/b)

[John W. Eaton]

| Yes, that fixed it, thanks!  Learning something new every day.
| 
| --Tim
| 
| 
| > -----Original Message-----
| > From: Ben Abbott [mailto:address@hidden 
| > Sent: Monday, April 05, 2010 5:01 PM
| > To: address@hidden
| > Cc: address@hidden
| > Subject: Re: precision of floor(a/b)
| > 
| > 
| > On Apr 5, 2010, at 7:52 PM, Tim Rueth wrote:
| > 
| > > If I do floor(6/1), I correctly get 6.  If I do 
| > floor(0.6/0.1), I get 5.  If I do floor((0.6/0.1 + 
| > 0.00000001)) I correctly get 6.
| > >  
| > > I assume this has to do with the precision of the division? 
| >  I read in the manual that Octave stores numbers as double 
| > precision, so this surprises me.  What's the best way to deal 
| > with this imprecision so I can always get the correct answer, 
| > or is something else going on here?
| > 
| > Will floor(0.6/0.1*(1+eps)) work for you?

I don't think this is a general solution to the problem.  It will
likely do the "right thing" (i.e., what you expect for decimal
arithmetic) in some cases but not others.  For example, what about

  floor (0.7999999999999999/0.1*(1+eps))

which on my system returns 8.  Note that in this case,
0.7999999999999999 < 0.8 does evaluate to true, so I guess you would
want the result of the division and floor operation to be 7, not 8 if
you were doing decimal arithmetic.

The thing is, you aren't doing decimal arithmetic.  So throwing in
factors of eps to get solutions that look right for decimal arithmetic
in certain cases is probably not what you want to do.

jwe