bug#22814: 25.0.91; Emacs runs out of file descriptors on OS X

[Anders Lindgren]

Hi!

I just gave this some thought (without knowing anything about the kqueue notifications).

I don't think the problem is what kqueue does when it run out of file descriptors, but how the rest of the Emacs process acts when this happens. For example, can it even read and write files? Can subprocesses be started?

The question, then, is how should the kqueue system work so that it doesn't run Emacs out of file descriptors? For example, if it would be possible to check how many are available, it could stop when there are, say, 50 remaining. Effectively, this would give a user 200 files that are auto-reverted using the notification system, and the rest would be handled by the old timer system.

    -- Anders

On Sat, Feb 27, 2016 at 2:26 PM, Michael Albinus <michael.albinus@gmx.de> wrote:

Eli Zaretskii <eliz@gnu.org> writes:

>> > One simple way to handle this is to define a variable with "max"
>> > number of files the notification system can handle. We can set this
>> > to, say, 200 on OS X and unlimited on other systems.
>>
>> Would be possible, yes. I would prefer to set the limit to a system
>> related value. Does there exist a portable way to detect, how many file
>> descriptors / processes Emacs is able to consume?
>
> Not portably, AFAIK.

On POSIX systems, we could propagate the result of

if !(getrlimit (RLIMIT_NOFILE, &rlim))
   return make_number (rlim.rlim_cur);
else
   return Qnil;

Maybe we add a function like `<library>-max-descriptors' to the
libraries. Or maybe not, and the backends check this themselves, and
cease to work when reaching an internal limit. Such an internal limit
could be half the number of soft RLIMIT_NOFILE on POSIX systems, for
example.

> Also, different implementations use different
> resources for receiving file notifications.  For example, w32notify
> uses one handle and one thread per watched file/directory.  The number
> of handles a process can have on Windows is very large, and the
> theoretical max number of threads is 32K, but both are limited by the
> resources already consumed by the Emacs process.  So determining the
> practical maximum at any given point will require a non-trivial
> function.

There must be different scenarios for different file notification
libraries. But using RLIMIT_NOFILE as basis when possible gives us
better results on systems like OS X and FreeBSD, which both use kqueue,
but provide different ressource limits.

Best regards, Michael.