[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Groff] About numerical precision
From: |
Ralph Corderoy |
Subject: |
Re: [Groff] About numerical precision |
Date: |
Wed, 02 Apr 2003 11:50:15 +0100 |
Hi Thomas,
> But, I can't manage to get \H'36432*72/46800' without any error:
> \H'\\nY*\\n(.s/\\nX' produces a multiplication overflow
>
> and \H'36432/46800*72' seems to produce some \H'0' for obvious
> precision reasons.
>
> Any idea ?
Since you know the numbers are up in the thousands, how about avoiding
the overflow by scaling them down first.
$ bc
36432*72/46800
56
36432/46800*72
0
(36432/100*72)/(46800/100)
56
Cheers,
Ralph.