[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Octave-bug-tracker] [bug #36133] num2str displays more than 16 signific
From: |
Rik |
Subject: |
[Octave-bug-tracker] [bug #36133] num2str displays more than 16 significant digits for large integer inputs |
Date: |
Wed, 25 Apr 2018 17:43:54 -0400 (EDT) |
User-agent: |
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:55.0) Gecko/20100101 Firefox/55.0 |
Follow-up Comment #10, bug #36133 (project octave):
Results from Matlab will be interesting, but I doubt that they test that each
number to be printed is an exact power of two. There are 1024 of those
numbers compared to 2^53 (9e15) numbers representable in the mantissa. It
really, really fails the 80/20 rule to worry about them.
According to Matlab's documentation for both sprintf and num2str, they are
aware of the issue
Note
If you specify a precision operator for floating-point values that exceeds the
precision of the input numeric data type, the results might not match the
input values to the precision you specified. The result depends on your
computer hardware and operating system.
Also, according to the documentation for num2str
s = num2str(A) converts a numeric array into a character array that represents
the numbers. The output format depends on the magnitudes of the original
values
That's fine, but my guess is that Matlab just uses a form of the %g format for
both floating point and integer numbers. Octave, however, tries to switch
format based on whether the input is an integer. The code is
if (any (x(valid) != fix (x(valid))))
## Floating point input
ndgt = max (ndgt + 5, 5); # Keep at least 5 significant digits
ndgt = min (ndgt, 16); # Cap significant digits at 16
fmt = sprintf ("%%%d.%dg", ndgt+7, ndgt);
else
## Integer input
ndgt += 3;
if (any (! valid))
ndgt = max (ndgt, 5); # Allow space for Inf/NaN
endif
## FIXME: Integers must be masked to show only 16 significant digits
## See test case for bug #36133 below
fmt = sprintf ("%%%d.0f", ndgt);
endif
I bet changing the integer case would fix some of this.
_______________________________________________________
Reply to this item at:
<http://savannah.gnu.org/bugs/?36133>
_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/