[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Octave-bug-tracker] [bug #56874] corr > 1
|
From: |
A.R. Burgers |
|
Subject: |
[Octave-bug-tracker] [bug #56874] corr > 1 |
|
Date: |
Mon, 9 Sep 2019 14:11:56 -0400 (EDT) |
|
User-agent: |
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/18.17763 |
Follow-up Comment #1, bug #56874 (project octave):
corr(x,y) - 1 = 2.2204e-16
numerical precision limit is hit in these lines:
c = cov (x);
s = sqrt (diag (c));
retval = c ./ (s * s');
results could maybe be constrained to [-1,1]
retval = min(max(c ./ (s * s'), -1), 1);
However the result could still be 1-eps. The user of corr needs to anticipate
that the result is subject to numerical errors.
_______________________________________________________
Reply to this item at:
<https://savannah.gnu.org/bugs/?56874>
_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/