bug#9264: 24.0.50; (file-error "Creating process pipe" "no error")

[Eli Zaretskii]

> Date: Fri, 23 Sep 2011 13:55:22 +0300
> From: William Xu <william.xwl@gmail.com>
> Cc: 9264@debbugs.gnu.org
> 
> It seems I'm able to reproduce it easily now.
> 
> 1. emacs -Q
> 2. eval below:
> 
> (progn
>   (dotimes (i 50)
>     (start-process "hi" nil "ls")
>     (message "%d" i)
>     ;(sleep-for 1)
>     ))
> 
> It gives an error:
>    while: Spawning child process: resource temporarily unavailable

I'm not sure the problem reproduced by this snippet is the same one as
what you reported originally.  In the above snippet, the problem
happens because we never give Emacs a chance to take note of the
processes that exit, and free the handles used for the 2 pipes we open
for each subprocess.  If I uncomment the sleep-for call, the program
runs to completion with no problems, even if I replace 1 with 0.1.

The underlying issue is that the Windows build of Emacs is limited to
31 simultaneous subprocesses.  That's because the APIs used on Windows
to listen to subprocesses are limited to 64 handles, and we use 2
handles per pipe (3 more handles are taken by the standard I/O
handles).  So we cannot start 50 subprocesses unless the first few
exit by the time we get to the 32nd process.  Emacs checks for exited
subprocesses when it is idle, but the above loop never gives it a
chance to do that.  Adding a call to sleep-for does, and so the
problem disappears.

I can achieve similar results with a patch I show below, which causes
sys_pipe to retry the failed _pipe call after doing the equivalent of
`(sleep-for 0.1)'.

However, I'm not sure this actually solves your original problem, for
two reasons:

  . you said that your problem starts happening only after some time
    that Emacs is up and running, whereas this recipe works right away
    after starting "emacs -Q"

  . I really doubt that you use some code that launches many
    subprocesses one after the other without any idleness in between

So I think there's a different bug somewhere.  Or maybe I'm missing
something.  Can you tell more about the context of your original
problem, which produced the following backtrace:

  Debugger entered--Lisp error: (file-error "Creating process pipe" "no error")
    call-process("ls")
    eval((call-process "ls") nil)
    eval-last-sexp-1(nil)
    ad-Orig-eval-last-sexp(nil)
    eval-last-sexp(nil)
    call-interactively(eval-last-sexp nil nil)

Was "ls" the only subprocess active at that time, or were you
launching many more at the same time?

If none of the above gives a clue, could you please add printf's to
the following functions:

 . create_child and register_child, where they assign cp->fd = fd

 . delete_child

In all of these places, please print cp->fd.  When the problem starts
to happen, it would be interesting to see which file descriptors
somehow were not released.

Here's the patch that allows your test case to run without failing:

=== modified file 'src/w32.c'
--- src/w32.c   2011-09-09 01:06:52 +0000
+++ src/w32.c   2011-09-29 15:37:54 +0000
@@ -5212,11 +5212,13 @@ sys_pipe (int * phandles)
 {
   int rc;
   unsigned flags;
+  int retried = 0;
 
   /* make pipe handles non-inheritable; when we spawn a child, we
      replace the relevant handle with an inheritable one.  Also put
      pipes into binary mode; we will do text mode translation ourselves
      if required.  */
+ retry:
   rc = _pipe (phandles, 0, _O_NOINHERIT | _O_BINARY);
 
   if (rc == 0)
@@ -5227,7 +5229,14 @@ sys_pipe (int * phandles)
        {
          _close (phandles[0]);
          _close (phandles[1]);
+         if (!retried)
+           {
+             wait_reading_process_output (0, 100000, 0, 0, Qnil, NULL, 0);
+             retried = 1;
+             goto retry;
+           }
          rc = -1;
+         errno = EAGAIN;
        }
       else
        {