Re: Is this a numerical problem?

[Jaroslav Hajek]

On Thu, Aug 19, 2010 at 2:32 PM, Lukas Reichlin
<address@hidden> wrote:
>
> On 19.08.2010, at 13:16, Jaroslav Hajek wrote:
>
>> On Thu, Aug 19, 2010 at 12:52 PM, Lukas Reichlin
>> <address@hidden> wrote:
>>> Hi Jaroslav
>>>
>>> I realized that (in most cases, see below) floats are rounded, not 
>>> truncated such that -4.00004 and -3.99995 are both displayed as -4.0000. 
>>> Therefore you can't tell whether -4.0000 is less or greater than -4.
>>>
>>> octave:15> 3.9999
>>> ans =  3.9999
>>> octave:16> 3.99995
>>> ans =  4.0000
>>> octave:17> 0.99995
>>> ans =  0.99995
>>> octave:18> 0.999995
>>> ans =  0.99999  # I'm expecting 1.00000 here
>>> octave:19> 0.9999995
>>> ans =  1.00000
>>> octave:20> 0.999996
>>> ans =  1.00000  # compare this to command 18
>>>
>>> Regards,
>>> Lukas
>>>
>>
>> Exactly. Which brings us back to your observation:
>>
>> octave:1> format long
>> octave:2> a = (10 * rem (-12/5, 1))
>> a = -4.00000000000000
>> octave:3> a - -4.00000000000000
>> ans =  8.88178419700125e-16
>> octave:4> -3.99999999999999 - a
>> ans =  9.32587340685131e-15
>>
>> which is an explanation.
>>
>> --
>> RNDr. Jaroslav Hajek, PhD
>> computing expert & GNU Octave developer
>> Aeronautical Research and Test Institute (VZLU)
>> Prague, Czech Republic
>> url: www.highegg.matfyz.cz
>
> Thanks for your patience. I consider the display of result 54 as an 
> inconsistency or a (cosmetic) bug because round (0.5) is 1. Rounding up 
> starts at 0.50, not at 0.51.
> (Only the first digit after truncation is significant for rounding)
>
> octave:57> 0.999996
> ans =  1.00000
> octave:53> 0.999995
> ans =  0.99999  # this should be 1.00000 like the previous result
> octave:54> 0.9999951
> ans =  1.00000
> octave:55> 0.9999950
> ans =  0.99999  # this should be rounded up too
> octave:56> 0.9999949
> ans =  0.99999
>
>
> Results from the "dark side":
>>> 0.99995
>
> ans =
>
>    1.0000  % this is correct
>
>>> 0.999949
>
> ans =
>
>    0.9999
>
>>> 0.999950
>
> ans =
>
>    1.0000
>
>>> 0.999951
>
> ans =
>
>    1.0000
>
>>> 0.99996
>
> ans =
>
>    1.0000
>
>>>
>
> Cheers,
> Lukas
>
>

If you think you've found a bug, please report it to the tracker.


-- 
RNDr. Jaroslav Hajek, PhD
computing expert & GNU Octave developer
Aeronautical Research and Test Institute (VZLU)
Prague, Czech Republic
url: www.highegg.matfyz.cz