[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Why does this happen

From: John W. Eaton
Subject: Why does this happen
Date: Tue, 13 Oct 1998 10:35:36 -0500 (CDT)

On 13-Oct-1998, Preben Randhol <address@hidden> wrote:

| I have octave 2.0.13 on a Linux 2.0.32 with RedHat 5.0 (glibc)
| I cannot understand why this happens:
| octave:2> B=[7973.6;.96500;2.7140]
| B =
|   7973.60000
|      0.96500
|      2.71400
| octave:3> format long
| octave:5> B
| B =
|   7973.600000000000364
|      0.965000000000000
|      2.714000000000000
| Why is B(1) now suddenly 7973.600000000000364 and not
| 7973.600000000000000 ?

The short answer is that the binary floating point format that your
computer uses to store numbers can only represent some decimal values
exactly.  In your example, you've asked for more digits than you have
bits to represent them 7973.6, and the formatting code in the C/C++
library is doing the best job it can given limited precision of the
binary floating point data.  Note that the other numbers are also not
represented exactly.  If you ask for more digits, you will see similar
results for all the numbers in your B vector:

  Octave, version 2.0.13 (i686-pc-linux-gnulibc1).
  Copyright (C) 1996, 1997, 1998 John W. Eaton.
  This is free software with ABSOLUTELY NO WARRANTY.
  For details, type `warranty'.

  octave:1> output_max_field_width = 40;
  octave:2> output_precision = 30;
  octave:3> B=[7973.6;.96500;2.7140]
  B =


Here is an example of a decimal fraction that can be represented

  octave:4> 1.25
  ans = 1.25000000000000000000000000000

You might argue that the binary to decimal conversion code should do
something different, but first you should probably read the code in
floatconv.c in the Linux C/C++ I/O library along with the paper that
is referenced there:

 Inspired loosely by William D. Clinger's paper "How to Read Floating
 Point Numbers Accurately" [Proc. ACM SIGPLAN '90, pp. 92-101].

Welcome to the wonderful world of floating point arithmetic.  :-)


reply via email to

[Prev in Thread] Current Thread [Next in Thread]