Re: Concurrency, again

[Ken Raeburn]

> On Oct 18, 2016, at 06:41, Eli Zaretskii <address@hidden> wrote:
> 
>> From: Ken Raeburn <address@hidden>
>> Date: Tue, 18 Oct 2016 06:08:49 -0400
>> Cc: address@hidden,
>> address@hidden
>> 
>>>> * header inclusion order requirement is weird; can generating one big 
>>>> header help?
>>> 
>>> I'm not sure I understand what inclusion order is being alluded to
>>> here.  Can you elaborate?
>> 
>> I think it’s the ordering and recursive inclusion involving thread.h 
>> relative to the other headers.  For example lisp.h includes thread.h which 
>> includes sysselect.h which includes lisp.h; thread.h uses struct 
>> vectorlike_header, so it has to be included in lisp.h after that structure 
>> is defined but before struct thread_state gets used; thread.h also includes 
>> regex.h which includes lisp.h.  You commented at one point, “We now have an 
>> unfortunate situation whereby lisp.h cannot be included before some of the 
>> other headers, due to this.”
> 
> This could be solved by moving parts of thread.h into lisp.h, with the
> appropriate #ifdef guards.  I think Lisp objects should be in lisp.h
> anyway, even if they are optional; anything else is confusing.

For big, specialized objects like buffers, I think the modularity can help keep 
things organized, but where we’ve got a structure with all the per-thread state 
from random parts of the program, like condition handlers and regex state, we 
don’t have the luxury of separating things.  But even if we move most of 
thread.h into lisp.h, I think there may still be mutual recursion between it 
and the other headers.

Perhaps if the thread state contained struct pointers instead of structures, we 
could forward-declare some of the types and not have to pull in the other 
headers in such a fashion.


> 
>>>> * one thread per terminal?
>>> 
>>> Why?
>>> 
>>>> * file notifications and such shouldn’t go through same queue as keyboard 
>>>> events
>>> 
>>> Why?
>> 
>> Stefan’s message: 
>> http://lists.gnu.org/archive/html/emacs-devel/2013-08/msg00755.html
> 
> I don't see that as a critical problem, perhaps because we don't yet
> realize how serious it can be.  The whole purpose of trying to merge
> the concurrency branch is to collect practical experience as to what
> should and shouldn't be in this kind of Emacs feature.  So I'd tend to
> let this be, until we find out we can't, and why.
> 
>>>> * interaction of SIGCHLD handling and threads?
>>> 
>>> Details?
>> 
>> Your message 
>> http://lists.gnu.org/archive/html/emacs-devel/2013-08/msg00738.html raised 
>> questions.  If they were ever satisfactorily answered, I overlooked it when 
>> putting together my notes….
> 
> That should be easy: since a subprocess is locked to a single thread,

by default, but if that thread exits, that lock disappears

> SIGCHLD should be delivered to that thread.  If we don't have that
> already, we should add that, it doesn't sound hard, given the
> infrastructure we already have (deliver_thread_signal etc.).

It’s not completely trivial.  Under POSIX, we get no control over which thread 
receives SIGCHLD due to a subprocess exiting, except if we want certain threads 
to block receiving the signal completely.  We can use pthread_kill to 
explicitly send SIGCHLD to a specific thread, if it’s not blocking the signal, 
but we don’t know which thread until we’ve fetched the child’s pid and looked 
it up in some data structure; we do that by calling waitpid() but then the 
kernel discards the child process’s status info, so the “correct” thread can no 
longer respond to the signal by making another waitpid() call to collect the 
status info.  We’d have to save the info the first time we call waitpid().  But 
doing it from within the signal handler could be tricky, because in that 
context we’re limited to async-signal-safe functions, and helpful routines like 
malloc() and pthread_mutex_lock() aren’t on the list.

On the other hand, perhaps we can create one special thread to do all the 
waitpid() calls and pass info to the Lisp-running threads.  If that’s all it’s 
doing, the non-signal-handler portion of the thread’s code can loop calling 
waitpid and locking mutexes and updating data structures, and the Lisp-running 
threads can check in their thread-state structures for messages from the 
child-reaping thread.  I’m not sure how safe it would be to block SIGCHLD in 
threads that might call into the various system UI libraries and such, but we 
can probably just use a simple SIGCHLD handler to let any thread wake the 
reaper thread, which can then do its waitpid() calls.


> 
>> It’s easy enough to disable stack overflow checking when enabling thread 
>> support.
> 
> Or add some simple code in the stack overflow handler to check if we
> are in the main thread, and if not, punt (i.e. crash).
> 
>> If only one thread is allowed into the image processing code at a time 
>> (i.e., don’t release the global lock for that code) then that’s probably 
>> fine for now, and there’s probably other state there that different threads 
>> shouldn’t be mucking around with in parallel.
> 
> Redisplay runs in the main thread anyway, right?  If so, there's no
> problem.

If some random thread calls (redisplay) or (sit-for …)?  I think it’ll run in 
whichever Lisp-running thread triggers it.  But, it’ll be the one holding the 
lock.

> 
>> The keyboard.c one is the only one I’m a bit concerned about, in part 
>> because I haven’t looked at it.
> 
> What part(s) of keyboard.c, exactly?

Anything looking at getcjmp; that means read_event_from_main_queue and 
read_char.  Like I said, I haven’t looked very closely; if the static storage 
isn’t ever used across a point where the global lock could be released to allow 
a thread switch, it may be fine.