[bug-gawk] gawk erroneous comparison with large negative exponents

[Sol Katzman]

Dear bug-gawk,

$ awk --version
GNU Awk 3.1.7

$ cat /etc/centos-release
CentOS release 6.6 (Final)

$ echo $ARCH
x86_64

It appears that large negative exponents in numerical input may cause comparison
operations to be treated as string compares, even though arithmetical operations
(or printf statements) with the same input do not cause underflow.

If I force an arithmetic operation (e.g. by adding 0 to the input) the 
comparison is correct.

The break between correct and incorrect comparison occurs at 1e-308 which is the
limit for double precision floating point.

In contrast, a perl script to do the same comparison seems to consistently 
underflow
to zero in the comparison and in arithmetic.

Below are examples and other supporting information.

Thanks for your attention,
/Sol Katzman

-----------------------------------------------------------------
AWK TEST SCRIPT THAT FAILS COMPARISON (BUT NO UNDERFLOW IN printf) AT OR BELOW 
1e-308:

foreach log10 (305 306 307 308 309 310 311)
    echo "1.0e-${log10}" | \
    awk '{if ($1 < 0.50) printf("awk  PASS"); else printf("awk  FAIL"); \
          printf("  %s:  %.10e\n",$1,$1)} '
end

awk  PASS  1.0e-305:  1.0000000000e-305
awk  PASS  1.0e-306:  1.0000000000e-306
awk  PASS  1.0e-307:  1.0000000000e-307
awk  FAIL  1.0e-308:  1.0000000000e-308
awk  FAIL  1.0e-309:  1.0000000000e-309
awk  FAIL  1.0e-310:  1.0000000000e-310
awk  FAIL  1.0e-311:  1.0000000000e-311

-----------------------------------------------------------------
PERL TEST SCRIPT THAT PASSES (WITH UNDERFLOW IN printf AT OR BELOW 1e-309):

foreach log10 (305 306 307 308 309 310 311)
    echo "1.0e-${log10}" | \
    perl -ne 'chomp; \
              if ($_ < 0.50) {printf "perl PASS";} else {printf "perl FAIL";} \
              printf "  %s:  %.10e\n",($_,$_); '
end

perl PASS  1.0e-305:  1.0000000000e-305
perl PASS  1.0e-306:  1.0000000000e-306
perl PASS  1.0e-307:  1.0000000000e-307
perl PASS  1.0e-308:  1.0000000000e-308
perl PASS  1.0e-309:  0.0000000000e+00
perl PASS  1.0e-310:  0.0000000000e+00
perl PASS  1.0e-311:  0.0000000000e+00

-------------------------------------------------
AWK TEST SCRIPT THAT PASSES COMPARISON (ADD 0 to string input):

foreach log10 (305 306 307 308 309 310 311)
    echo "1.0e-${log10}" | \
    awk '{if ($1+0 < 0.50) printf("awk+0  PASS"); else printf("awk  FAIL"); \
          printf("  %s:  %.10e\n",$1,$1+0)} '
end

awk+0  PASS  1.0e-305:  1.0000000000e-305
awk+0  PASS  1.0e-306:  1.0000000000e-306
awk+0  PASS  1.0e-307:  1.0000000000e-307
awk+0  PASS  1.0e-308:  1.0000000000e-308
awk+0  PASS  1.0e-309:  1.0000000000e-309
awk+0  PASS  1.0e-310:  1.0000000000e-310
awk+0  PASS  1.0e-311:  1.0000000000e-311

----------------------------------------------------
R CODE SHOWING VALUE OF EXPONENTS:

> ex <- 305:311
> sm <- 10^-ex
> log2sm <- log2(sm)
> ex.sm.log2sm <- cbind (ex,sm,log2sm)
> ex.sm.log2sm
      ex     sm    log2sm
[1,] 305 1e-305 -1013.188
[2,] 306 1e-306 -1016.510
[3,] 307 1e-307 -1019.832
[4,] 308 1e-308 -1023.154
[5,] 309 1e-309 -1026.476
[6,] 310 1e-310 -1029.798
[7,] 311 1e-311 -1033.120
>

----------------------------------------------------
TABLE OF PRECISION FROM:

  
https://www.gnu.org/software/gawk/manual/html_node/Floating_002dpoint-Context.html#Floating_002dpoint-Context

Name    Total bits Precision     emin    emax
Single          32      24       -126    +127
Double          64      53      -1022   +1023
Quadruple      128     113     -16382  +16383

"In case an underflow occurs, the standard allows, but does not require, the result from an arithmetic operation to be a number 
smaller than the smallest nonzero normalized number. Such numbers do not have as many significant digits as normal numbers, and 
are called denormals or subnormals. The alternative, simply returning a zero, is called flush to zero. The basic IEEE-754 binary 
formats support subnormal numbers."