Re: Output precision

[Richard Balogh]

JuanPi wrote:
> Hi,
> Could anybody give me a help understanding this?
>
> > x = 987654321.123456789;
> x =  9.8765e+08
> > output_precision(45); output_max_field_width(50)
> > x
> x =  987654321.123456835746765136718750000000000000
>
> Why is x not displayed in the way I created? In other words, Where did
> the last couple of digits go?
>
> Another results that puzzles me is (using the output precision and
> field width as above)
> > x = 987654321.123456
> x =  987654321.123456001281738281250000000000000000
>
> Thank you very much.

Let's look on the simpler example:

0.5(DEC) = 0.1 (BIN)
0.25(DEC) = 0.01 (BIN)
but, for example
0.1(DEC) = 0.0001100110011001100110011001100110011001100110011001101... (BIN)

- with the limited precision the numbers cannot be represented exactly.
For your example, last couple of digits is rounding error. This is the
exact value of the closest number represented with 50 binary bits.

Richard Balogh