Re: "resource temporarily unavailable" errors on windows 7

[Eli Zaretskii]

> Date: Tue, 13 Mar 2012 12:50:09 +0800
> From: Alex Harsanyi <address@hidden>
> Cc: address@hidden
> 
> 2012/3/13 Eli Zaretskii <address@hidden>:
> > [Please keep the list on the CC.]
> >
> >> Date: Tue, 13 Mar 2012 09:31:18 +0800
> >> From: Alex Harsanyi <address@hidden>
> >>
> >> Just a quick update: I modified allocate_heap() to start asking for
> >> 1.5GB initially and that also seemed to fix the subprocess creation
> >> problem.
> >
> > What value exactly did you use?
> 
>   unsigned long size = 0x60000000; /* start by asking for 1.5GB */

Is this the largest value that gives you a fully functional Emacs?  Or
can you enlarge it some more, before subprocesses start failing?  I'm
not happy reducing the maximum memory possible by 0.5GB, especially
since no one else complained about this yet, including people who run
Emacs on Windows 7.  (But I still hope that we can find a better
solution than just reducing the memory request; see below.  So this is
for now a fallback.)

> Since I have more memory available than Emacs requests, I don't think
> enlarging the amount of virtual memory would help.  Starting a new
> Emacs session does not seem to affect the amount of free memory
> available significantly: my Emacs session uses 25Mb of physical memory
> after start-up.  My main Emacs session (where I do my work) currently
> uses 63Mb, not a very large value.  The 1.5GB "reservation" by
> allocate_heap() does not decrease the amount of available virtual
> memory.

OK, good.  This seems to indicate that the problem is with the memory
of the Emacs process itself, not with the memory for other processes,
such as the subprocess you are trying to launch.  Most probably,
something we do during spawning the subprocess tries to reserve
memory, similarly to what allocate_heap does, and fails because the
sum of what's already reserved and this new reservation exceeds 2GB.

> Perhaps I was a bit imprecise with the terminology, but I did realise
> that allocate_heap() only reserves the memory and that the 2GB address
> space limit is per process, not per system.

Sorry, I couldn't be certain that you understand all the details about
that function.

> There seems to be some documentation on the net that ANSI
> CreateProcess will allocate memory to convert the command line to
> UNICODE before calling the UNICODE version of CreateProcess, this is
> what I meant by "CreateProcess fails to allocate memory". Could this
> memory  allocation fail if all the memory is already reserved for
> something else?

It could in theory, but I very much doubt that conversion of any
reasonable command line to UTF-16 could ever need so much memory
(0.2GB, the difference between the 1.8GB allocation you reported and
the 2GB limit).

Anyway, we have no evidence that the error comes from the
CreateProcess API.  On the contrary (see below), I think we have
evidence that it comes from elsewhere.

I thought about this some more, read some stuff on MSDN and elsewhere,
and I have the following questions and observations.

First, the error message.  It is triggered by this code in
w32proc.c:sys_spawnve:

  cp = new_child ();
  if (cp == NULL)
    {
      errno = EAGAIN;
      return -1;
    }

Now, in new_child we see this:

    cp->char_avail = CreateEvent (NULL, TRUE, FALSE, NULL);
    if (cp->char_avail)
      {
        cp->char_consumed = CreateEvent (NULL, FALSE, FALSE, NULL);
        if (cp->char_consumed)
          {
            cp->thrd = CreateThread (NULL, 1024, reader_thread, cp, 0, &id);
            if (cp->thrd)
              return cp;
          }
      }
    delete_child (cp);
    return NULL;
  }

So this can happen due to error either in one of the two calls to
CreateEvent or in the call to CreateThread.  My crystal ball bets on
the latter, but could you please verify that this is the problem?  If
it is, please add a call to GetLastError there and see which error
code is returned.

Assuming it's CreateThread that fails, the obvious suspicion would be
that it fails because creating a thread reserves some memory for its
stack.  The call above is careful to override the amount of stack
memory that is actually _committed_ and sets it at a mere 1KB, but the
memory _reserved_ for the stack is still 1MB, according to MSDN.

Question: does the problem happen right when you invoke the first
subprocess, or only after several subprocesses are already running,
and you try to invoke another one?

Also, can you use the VMMap utility (from Sysinternals) and tell what
it shows in the "Private Data", "Thread Stack", and "Free" categories,
when you run Emacs with the original code in allocate_heap, after you
get the error message?  Can you spot, in particular, how much memory
is reserved  and committed for Emacs and for the additional thread it
starts in new_child?

TIA