powerpc64le-linux long double math test failures

[Ulrich Weigand]

Hello,

with the new little-endian 64-bit PowerPC platform we're seeing failures in
some of the long double math tests.

PowerPC uses the IBM "double double" format as long double data type, which
consists of a pair of 64-bit IEEE double values; the value of the long
double is the sum of the two doubles.  Special values (inf, nan) are
represented in the first double of the pair, in which case the second
double is ignored.

The little-endian representation of this data type still uses a pair of
doubles, and the more significant is still the first element of the pair.
The two doubles themselves, however, now use the little-endan 64-bit IEEE
encoding.

The failing tests are the long double versions of test-isinf, test-isnanl
etc.   In particular, this construction doesn't work as expected:

#if defined LDBL_EXPBIT0_WORD && defined LDBL_EXPBIT0_BIT
  /* A bit pattern that is different from a Quiet NaN.  With a bit of luck,
     it's a Signalling NaN.  */
  {
    memory_long_double m;
    m.value = NaNl ();
# if LDBL_EXPBIT0_BIT > 0
    m.word[LDBL_EXPBIT0_WORD] ^= (unsigned int) 1 << (LDBL_EXPBIT0_BIT -
1);
# else
    m.word[LDBL_EXPBIT0_WORD + (LDBL_EXPBIT0_WORD < NWORDS / 2 ? 1 : - 1)]
      ^= (unsigned int) 1 << (sizeof (unsigned int) * CHAR_BIT - 1);
# endif
    m.word[LDBL_EXPBIT0_WORD + (LDBL_EXPBIT0_WORD < NWORDS / 2 ? 1 : - 1)]
      |= (unsigned int) 1 << LDBL_EXPBIT0_BIT;
    ASSERT (isnanl (m.value));
  }
#endif

This starts out with a quiet NaN and flips the top mantissa bit, making it
a signalling NaN unless the mantissa is now zero.  To prevent that, the
last instruction tries to ensure at least one (but not the top) mantissa
bit is set.

And this is what doesn't work on powerpc64le.  This algorithm in general is
dubious for the IBM double double format, since it assumes a simple layout
where the mantissa follows the exponent.  On big-endian systems, the
algorithm happens to work anyway, since LDBL_EXPBIT0_WORD is zero.  The
code therefore operates on the first two words, which are actually in
64-bit IEEE double representation, and thus the code transforms a 64-bit
IEEE qNaN into a corresponding signalling NaN --- as the second double is
ignored for NaNs, this therefore also works correctly for long double.

However, in the little-endian format, LDBL_EXPBIT0_WORD is 1.  Due to the <
NWORDS / 2 test, the last instruction above now sets a bit in word 2, which
belongs to the second double.  The first double therefore retains a zero
mantissa and the whole long double therefore represents infinity, not a
NaN.

I'm wondering now what the best way to fix this would be.  I'm a little bit
confused about the original intentions of the test, however.  Why does it
attempt to move to another word?  If it simply attempted to set the
*second* highest mantissa bit, everything would work out OK.   And if it
does move to another word, why the NWORDS / 2 test?  Should we trust the
gl_BIGENDIAN macro instead?

Or should the test simply hardcode the information about the IBM double
double format, and treat the first half as 64-bit IEEE double?

I'd be happy to come up with a patch, but I'd appreciate some guidance what
direction it should take.


Mit freundlichen Gruessen / Best Regards

Ulrich Weigand

--
  Dr. Ulrich Weigand | Phone: +49-7031/16-3727
  STSM, GNU/Linux compilers and toolchain
  IBM Deutschland Research & Development GmbH
  Vorsitzende des Aufsichtsrats: Martina Koederitz | Geschäftsführung: Dirk
Wittkopp
  Sitz der Gesellschaft: Böblingen | Registergericht: Amtsgericht
Stuttgart, HRB 243294