Re: t-test gets NaN instead of 0 for significance on x86-64

[Ben Pfaff]

John Darrington <address@hidden> writes:

> On Thu, Sep 03, 2009 at 08:10:31PM -0700, Ben Pfaff wrote:
>      John Darrington <address@hidden> writes:
>      
>      > As a short term solution, the best I can suggest is that we 
>      > clamp pow2(correlation) to 1.0.
>      
>      Thanks.  I pushed out this stopgap fix.
>
> Isn't there still a problem when pairs[i].correlation is less than
> -1 ?  For example -1.00000001

You are right, of course.

Could you check that this re-fix looks correct?

commit 46912334775c83a23902bd5c7d72cd4ad8d23c95
Author: Ben Pfaff <address@hidden>
Date:   Fri Sep 4 09:05:08 2009 -0700

    T-TEST: Also fix the case where the correlation is slightly less than -1.0.
    
    Thanks to John Darrington for pointing out the omission.

diff --git a/src/language/stats/t-test.q b/src/language/stats/t-test.q
index dd40de9..ba7e938 100644
*** a/src/language/stats/t-test.q
--- b/src/language/stats/t-test.q
***************
*** 1432,1447 ****
  
        double df = pairs[i].n -2;
  
!       /* pairs[i].correlation is a correlation, so mathematically it will
!          always be in the range [-1.0, 1.0].  Inaccurate calculations 
sometimes
!          cause it to be slightly greater than 1.0, however, which makes the
!          sqrt() below to come out as NaN instead of 0.  So force it to be 1.0
!          or less. */
!       double corr = MIN (1.0, pairs[i].correlation);
! 
        double correlation_t =
        pairs[i].correlation * sqrt (df) /
!       sqrt (1 - pow2 (corr));
  
  
        /* row headings */
--- 1432,1444 ----
  
        double df = pairs[i].n -2;
  
!       /* corr2 will mathematically always be in the range [0, 1.0].  
Inaccurate
!          calculations sometimes cause it to be slightly greater than 1.0, so
!          force it into the correct range to avoid NaN from sqrt(). */
!       double corr2 = MIN (1.0, pow2 (pairs[i].correlation));
        double correlation_t =
        pairs[i].correlation * sqrt (df) /
!       sqrt (1 - corr2);
  
  
        /* row headings */

-- 
Ben Pfaff 
http://benpfaff.org