Re: yet another Java oddity

[Mike Miller]

On Thu, Feb 7, 2013 at 3:14 AM, CdeMills wrote:
> This is not the right approach. "1e-5", as represented in IEEE
> double-precision arithmetic, is not a finite number, so it get somewhat
> truncated. The way this truncation is performed is out of reach. The correct
> test is abs(1e-5 - 10^-5) <= eps
>
> Same thing for the other one: 2.72 is of finite length in base 10, but not
> in base 2.

Yes, well aware of floating point representation. Regardless glibc's
libm *does* return the exact binary significand such that 1e-5 ==
std::pow(10, -5), and some of the tests in Octave have been relying on
that until this point. We can certainly change some of the tests to add
tolerances, which falls under what I said:

>> Either by addressing the underlying bug/feature or
>> by tweaking those particular routines that are failing.

However, here's a more serious example of a failure that at a glance
should have nothing to do with floating point precision:

octave:1> format native-hex;
octave:2> x = bitshift (uint64 (1), 63) - 1
x = ffffffffffffff7f
octave:3> javaObject ("java.lang.String");
octave:4> y = bitshift (uint64 (1), 63) - 1
y = 0000000000000080
octave:5> y = bitshift (uint64 (1), 63) - 512
y = 0000000000000080
octave:6> y = bitshift (uint64 (1), 63) - 513
y = 00fcffffffffff7f

The tests that are failing for me are chop, axis, betainv, normpdf, and
dec2base. I looked into them enough that I believe all of them are
failing because the value returned by std::pow is different when Java is
running.

On a related note, running "fntests" does not show any failures on most
of my test systems. I only get a failure with "test chop", "test
betainv", etc. Why would that be different?

-- 
mike