Re: [bug-gawk] float conversion

[Nelson H. F. Beebe]

The posted example code

        % gawk 'BEGIN { for (i = -1; i <= 1; i += 0.1) { print "int("i") = " 
int(i)} }'

illustrates once again the confusion felt by humans who are not
experts in floating-point arithmetic, and thus, often do not realize,
or have forgotten, that many computers work in a different number base
than humans do.

With the default arithmetic used in most current scripting languages
(awk, perl, python, php, ruby, ...), computations are done in the
binary floating-point system defined by the IEEE 754 Standards (1985
and 2008).  Thus, increments in loops should be chosen to be either
exactly representable in both binary and decimal, or as powers of two,
if arithmetic of simple addition is to be exact.  This variant
produces the humanly-expected output:

        % gawk 'BEGIN { for (i = -1; i <= 1; i += 1/8) { print "int("i") = " 
int(i)}}'
        int(-1) = -1
        int(-0.875) = 0
        int(-0.75) = 0
        int(-0.625) = 0
        int(-0.5) = 0
        int(-0.375) = 0
        int(-0.25) = 0
        int(-0.125) = 0
        int(0) = 0
        int(0.125) = 0
        int(0.25) = 0
        int(0.375) = 0
        int(0.5) = 0
        int(0.625) = 0
        int(0.75) = 0
        int(0.875) = 0
        int(1) = 1

I have versions of hoc and lua that work with decimal floating-point
arithmetic, and the rexx and netrexx scripting languages work entirely
in high-precision decimal arithmetic.  Here is a similar rexx example:

        % rexx
        y = -1
        do while  y <= 1
            say 'trunc(' y ') = ' trunc(y)
            y = y +  2 / 10
        end
        trunc( -1 ) =  -1
        trunc( -0.8 ) =  -0
        trunc( -0.6 ) =  -0
        trunc( -0.4 ) =  -0
        trunc( -0.2 ) =  -0
        trunc( 0 ) =  0
        trunc( 0.2 ) =  0
        trunc( 0.4 ) =  0
        trunc( 0.6 ) =  0
        trunc( 0.8 ) =  0
        trunc( 1.0 ) =  1

The decimal hoc equivalent looks like this:

        % hocd32
        for (i = -1; i <= 1; i += 0.2) { println "int(", i, ") = ", int(i)}
        int( -1 ) =  -1
        int( -0.8 ) =  -0
        int( -0.6 ) =  -0
        int( -0.4 ) =  -0
        int( -0.2 ) =  -0
        int( 0 ) =  0
        int(  0.2 ) =  0
        int(  0.4 ) =  0
        int(  0.6 ) =  0
        int(  0.8 ) =  0
        int(  1 ) =   1

Notice that hoc and rexx take care to produce the correct sign of the
integer part of negative arguments.  Many other languages are rather
sloppy in that regard.

lua does not have a native function equivalent to int(): here is
something close by, using floor():

        % dlua
        function printf(...)
         io.write(string.format(...))
        end

        y = -1
        while (y <= 1) do
            printf("math.floor(%g) = %g\n", y, math.floor(y))
            y = y + 0.2
        end

        math.floor(-1) = -1
        math.floor(-0.8) = -1
        math.floor(-0.6) = -1
        math.floor(-0.4) = -1
        math.floor(-0.2) = -1
        math.floor(0) = 0
        math.floor(0.2) = 0
        math.floor(0.4) = 0
        math.floor(0.6) = 0
        math.floor(0.8) = 0
        math.floor(1) = 1

-------------------------------------------------------------------------------
- Nelson H. F. Beebe                    Tel: +1 801 581 5254                  -
- University of Utah                    FAX: +1 801 581 4148                  -
- Department of Mathematics, 110 LCB    Internet e-mail: address@hidden  -
- 155 S 1400 E RM 233                       address@hidden  address@hidden -
- Salt Lake City, UT 84112-0090, USA    URL: http://www.math.utah.edu/~beebe/ -
-------------------------------------------------------------------------------