Re: precision of calculations

[A. Scottedward Hodel]

double precision arithmetic has about 16 significant digits
(depending on hardware).  Hence, if (A-B) should be zero, but
you get
    norm(A-B)/norm(A) approximately = 1e-16
you're o.k.  The relative error can creep up as problem dimension
and problem condition (how sensitive the results are to small
perturbations in data) increases.

For more information, consult Golub and Van Loan's book,
Matrix Computations, Johns Hopkins University Press, available
in paperback (for the frugal) or hardback (for those who use it 
a lot).

A S Hodel Assoc. Prof. Dept Elect Eng, Auburn Univ,AL  36849-5201
On leave at NASA Marshall Space Flight Center (256) 544-1426
Address until 15 Mar 2000:Mail Code TD-55, MSFC, Alabama, 35812
http://www.eng.auburn.edu/~scotte

----------
>From: Robert Butora <address@hidden>
>To: address@hidden
>Subject: precision of calculations
>Date: Fri, Jul 23, 1999, 7:27 AM
>

>Does anybody know how the precision of calculation 
>(not output_precision !) is controlled in Octave ?
>
>I've calculated some matrix where I expected results
>to be 1 or 0. Instead of 0 I've got values in order
>of 1e-16 ... 1e-18. 
>
>Should I chase an error in my calculation or does it
>have something to do with precision of calc. ?
>
>Could anybody give some general thoughts on precision in Octave ?
>
>Thanks 
>Robert
>
>
>
>---------------------------------------------------------------------
>Octave is freely available under the terms of the GNU GPL.  To ensure
>that development continues, see www.che.wisc.edu/octave/giftform.html
>Instructions for unsubscribing: www.che.wisc.edu/octave/archive.html
>---------------------------------------------------------------------
>




---------------------------------------------------------------------
Octave is freely available under the terms of the GNU GPL.  To ensure
that development continues, see www.che.wisc.edu/octave/giftform.html
Instructions for unsubscribing: www.che.wisc.edu/octave/archive.html
---------------------------------------------------------------------