[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

RE: [Groff] About numerical precision

From: Ted Harding
Subject: RE: [Groff] About numerical precision
Date: Wed, 02 Apr 2003 11:44:19 +0100 (BST)

On 02-Apr-03 Thomas Baruchel wrote:
> I have very easely (X-x)/3+x with a good precision in scaled points.
> For instance, calling my macro with Minion-Regular 72,
>   having x = 31248
>          X = 46800
>   I get (X-x)/3+x = 36432   (ie. 36.432p)
>   which is what I expected
> But, I can't manage to get \H'36432*72/46800' without any
> error: \H'\\nY*\\n(.s/\\nX' produces a multiplication overflow
> and \H'36432/46800*72' seems to produce some \H'0' for obvious
> precision
> reasons.
> Any idea ?

I just did

  .nr Y 36432
  .nr X 46800
  .nr N \nY*\n(.s/\nX

in groff 1.16.1 and got "56" printed out, with no overflow problems.
(Though perhaps you are using the "devdvi" device, whose device-unit
unitwidth is 131072, as opposed to 1000 for "devps" which I was using

One possible approach to deal with overflow could be to use something

  .nr N ((\nY+5)/10)*\n(.s/((\nX+5)/10)

which will reduce the size of the integer produced by the multiplication
(and, given the sizes of integers corresponding to device units, will
not cause perceptible error in the printed output, even though the
division will drop the "remainder").

Best wishes,

E-Mail: (Ted Harding) <address@hidden>
Fax-to-email: +44 (0)870 167 1972
Date: 02-Apr-03                                       Time: 11:44:19
------------------------------ XFMail ------------------------------

reply via email to

[Prev in Thread] Current Thread [Next in Thread]