bug#18420: 24.3; interaction with external process hangs emacs

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

> Did you try replacing accept-process-output with sit-for?

The current code has:

    (accept-process-output process 0.1)

This waits up to 0.1 seconds for output from the child process
ada_mode_wisi_parse (or any other process, but there are none in this
setup). In practice in this scenario, it returns much sooner than 0.1
seconds; there is some output very quickly after sending any part of
the buffer contents.

Replaced with:

    (sit-for 0.1)

This performs redisplay, then waits for SECONDS seconds or until "input"
is available. The wait is done with 'read-event', which reads from "the
input stream". So apparently this means user input, not child process
input.

This fixed the hang. There is no user input during this process; it's
too fast. The test buffer is 7274 bytes long, so there are three cycles
in the send-parse loop. Apparently what happens is each cycle
waits for 0.1 seconds, giving the child process enough time to process
all of that input. Then Emacs wakes up, processes all of the child
process output, resumes send-parse, and sends the next chunk.

This works, but is essentially a race condition between the sit-for
timer and the child process. It will be too slow for general use; I have
clients with 250,000 byte buffers, which would take 12.2 seconds (slower
than the current elisp parser). Obviously I can reduce the 0.1, but that
runs into the race condition and the deadlock. 0.01 would be fast
enough for the largest buffer, so I tried:

    (sit-for 0.01)

This hangs, in the same deadlock described above. So we need a fix for
the deadlock.

I can use (sit-for 0.1) as a workaround while I continue debugging the
rest of the problems I have, on reasonably sized buffers.

There is another workaround I can use; queue all output in
ada_mode_wisi_parse until all input is read and parsing is done, then
send it all. That would lose parallel processing time, which is
significant with today's processors. But I'll use it if we can't fix
this soon.

> How about enlarging the 2nd argument to accept-process-output -- did
> you try that, and if so, did it have any effect?

tried:

    (accept-process-output 1.0)

This hangs. Since (sit-for 0.1) allows enough time for the child process
to process all of one chunk, this is expected.

>> The loop is replacing an earlier version that used a single
>> process-send-string to send the entire buffer string. I was hoping that
>> the explicit accept-process-out calls would allow C-g to work, but
>> apparently not.
>
> I cannot easily interpret this, too much info is missing.  For
> starters, if the parser gets only part of its expected input (whose
> size you are apparently passing via command-line arguments), would it
> start parsing, or will it wait for the rest?

It starts parsing immediately, and generates some output quickly. The
byte count indicates when to stop parsing.

> If the latter, then the
> way you've broken the buffer string into chunks will not change
> anything, right?

It doesn't change the child process behavior. I was hoping it would
change the emacs side, allowing it to at least check for C-g every 0.1
seconds. Hmm, I'm not clear that accept-process-input checks for input
events. Perhaps we need a combined sit-for/accept-process-input?

>> The external process implements an LALR parser for Ada source; it
>> outputs the parse results back to Emacs. There is a lot of output, so
>> it can easily fill up the IO queue. Attaching to that process when
>> Emacs is hung shows it is blocked writing to stdout, in a normal part
>> of the code flow.
>
> Can you tell how much did it write to the pipe at that point?

No, that info is not tracked in the current code. I can add it.

The parser has read 2935 bytes.

However, there is some variation in precisely when the hang occurs; the
wisi-ada-parse-send-parse loop outputs the sent bytes each cycle, and
that number changes with each run (among the 3 possible choices for this
size buffer).

> What is your versions of GCC, Binutils, MinGW runtime, and w32 API
> headers, btw?

gcc for Emacs: gcc.exe (Rev3, Built by MSYS2 project) 4.9.1

binutils: 2.24.78559.d43808f-1 (msys2 package)

MinGW: not sure what this means; the runtime is just a bunch of
packages. For one: libgc 7.2.d-1

w32 API headers:

/msys64/mingw32/i686-w64-mingw32/include/windows.h has no version number

mingw-w64-i686-headers-git 4.0.0.4210.0177153-1

Hope that helps.

--
-- Stephe