Re: [bug-gawk] pi in gawk

[John Haque]

On Fri, Feb 17, 2012 at 08:39:48AM -0700, Nelson H. F. Beebe wrote:
...
> >> 
> >> $ ./gawk -M -vPREC=115 'BEGIN { printf("%.34f\n", atan2(0, -1)) }'
> >> 3.1415926535897932384626433832795029
> >> 
> >> However, rounding towards zero gives us:
> >> 
> >> $ ./gawk -M -vPREC=115 -vRNDMODE="RNDZ" 'BEGIN { printf("%.34f\n", 
> >> atan2(0, -1)) }'
> >> 3.1415926535897932384626433832795027
> >> 
> >> I was expecting 8 or 9 in the last digit, not 7.
> >> Is there an explanation?
> >> ...
> 
> Yes, there is an explanation: $\pi$ is a transcendental constant, and
> thus has an infinite expansion in digits of almost any base, except a
> base of $\pi$ itself, and powers thereof.  Thus, to represent it in
> human-readable decimal form, computation is required, and that
> involves inexact operations, each of which must be rounded to working
> precision according to some rounding rule.  The cumulative affect of
> several such operations almost always makes the last reported digits
> uncertain.  If precision is variable, as it is with the MPFR library,
> you can always work with a few more digits and mask those errors.  If
> you use n more bits that you finally want, then the chances of a
> worst-case rounding error that makes the last reported digit off by
> one goes like 2**(-n).  If n is large enough, that error is unlikely
> to be seen in practice.

Thanks for the explanation. It makes perfect sense.

The loss of precision happens during intermediate calculations
in the MPFR implementation of atan2, not because of any
computations done by me. In this case, both of the arguments of
atan2 are exactly representable. If the intermediate computations
in atan2 were done with a precision slightly higher than the
max of the precisions of the operands, then the off by one error
in the last digit of the result could have been avoided.
I guess it is the responsiblity of the user of the library.

Regards,

John