|
From: | Scott Carter |
Subject: | Re: precision of floor(a/b) |
Date: | Mon, 05 Apr 2010 17:11:44 -0700 |
User-agent: | Thunderbird 2.0.0.24 (Windows/20100228) |
This has nothing to do with Octave. Neither 0.6 nor 0.1 are exactly
representable in IEEE floating point, and as it turns out the ratio of
their model numbers is just a hair less than 6. So, according to IEEE arithmetic, as you might have guessed, 5 is indeed the "correct" answer to floor(0.6/0.1). I don't know what you are trying to do, so I don't know what the "correct" answer for what you are trying to do it (i.e., in what fashion do you know that your 0.6 is an exact value rather that 0.6 +/- some epsilon, so that the floor() function is what you want? ). If your inputs are in decimal notation and you believe that they represent "exact" values and are not especially large, you can multiply them by a suitable power of ten *before* converting to floating point. All the integers of reasonable value are exactly representable, and then the floor function will give you the correct value for those exact decimal fractions. Regards, Scott Tim Rueth wrote:
-- Scott Carter Sr. Manager Product R&D NKUS Lab Nihon Kohden Corp. Tel: 949-474-9207, X. 103 FAX: 949-474-8267 address@hidden |
[Prev in Thread] | Current Thread | [Next in Thread] |