Re: [Bug-apl] noise in fomatting ver small numbers

[alexweiner]

fair enough. good thing I only wanted to format to the one that "works" 

-------- Original Message --------
Subject: Re: [Bug-apl] noise in fomatting ver small numbers
From: Kacper Gutowski <address@hidden>
Date: Feb 21, 2016 5:46 PM
To: address@hidden
CC: "address@hidden" <address@hidden>

On Sun, Feb 21, 2016 at 10:38 PM,  <address@hidden> wrote:
> it seems that format adds fuzz after a certain point.

It's not a “fuzz,” but an artifact of using IEEE 754 double precision
floating point numbers.  ISO APL leaves a lot of freedom when it comes to
representation of numbers, but when using IEEE floats, the results you
observed are expected and anything else would be incorrect.  You should be
able to observe exactly the same results in any language using floating
point numbers.

>       a←0.00000000000000000000000000000000000000000001

The number 1E¯44 you requested isn't representable exactly, and the closest
representable one is a←8034690221294951×2⋆−199.

>       '0.000000000000000000000000000000000000000000000000000000000000 '⍕a
> 0.000000000000000000000000000000000000000000010000000000000000
>       '0.0000000000000000000000000000000000000000000000000000000000000 '⍕a
> 0.0000000000000000000000000000000000000000000099999999999999995

Format simply prints out its decimal expansion.  But it properly rounds
that expansion using the usual rules: 99.5 rounds up to 100, but 95.2
rounds down to 95.

-k