[Octave-bug-tracker] [bug #35827] The hygepdf and hygecdf functions are numerically unstable for large populations

[anonymous]

URL:
  <http://savannah.gnu.org/bugs/?35827>

                 Summary: The hygepdf and hygecdf functions are numerically
unstable for large populations
                 Project: GNU Octave
            Submitted by: None
            Submitted on: Tue 13 Mar 2012 01:19:38 PM UTC
                Category: Libraries
                Severity: 3 - Normal
                Priority: 5 - Normal
              Item Group: Incorrect Result
                  Status: None
             Assigned to: None
         Originator Name: Joachim Arts
        Originator Email: address@hidden
             Open/Closed: Open
         Discussion Lock: Any
                 Release: 3.6.1
        Operating System: Microsoft Windows

    _______________________________________________________

Details:

The hypergeometric pdf and cdf return NaN for large populations.
For example hygecdf(10,10000,200,1000) returns NaN, when the correct result,
also given in MATLAB is 0.007602821895.
Fixing by returning the binomial approximation for large populations is
dangerous here because for the example above, the binomial approximation is
binocdf(10,1000,200/10000)=0.010237. Thus this approximation can be quite poor
already.




    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?35827>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/