Re: Multithreading, again and again

[Stefan Monnier]

Stefan> I also recently noticed that sometimes my Gnus is unresponsive for
Stefan> a while (waiting for a remote server, maybe in DNS, tho I think not) and
Stefan> while C-g does work, other things don't, most importantly clipboard
Stefan> requests get delayed so if my Gnus session happens to own the clipboard
Stefan> then C-y in another Emacs session hangs until the remote server replies.
> I have noticed this as well.  Annoyingly, it makes my ERC connections
> time out.  I guess whatever is pausing is not properly accepting process
> output or processing timers or something like that.
> I was blaming opportunistic TLS for this but I didn't actually
> investigate.

Maybe you're right.  That deserves a separate bug report.

Tom> The other is the multiple keyboard case.  I think it should be possible
Tom> to have each 'emacsclient -t' run in a separate thread.
Stefan> This OTOH is pretty much "the general case" since it needs to deal with
Stefan> running Elisp in different threads concurrently (and with recursive
Stefan> edits and minibuffers, we want to be able to run one terminal's command
Stefan> while some other terminal still has active let-bindings).
> I think it is doable, given the context-switching constraints I
> mentioned earlier.

Yes a "simple" switch-on-yield should be sufficient (i.e. the old
concurrency branch should be able to handle it), but that still means
that solving such a problem requires handling interaction between the
various kinds of Elisp variable bindings (thread-local, buffer-local,
...).

> The long-term thing that would be cool is context-switching during QUIT.

I'm not sure it would be significantly easier than true concurrency.

> I don't think unconstrained context-switching is possible without some
> drastic changes to the implementation.

Why do you think so?

Tom> My plan was to follow the Bordeaux threads API, more or less.

Stefan> I'm afraid that using locks is not going to work well for Elisp.
Stefan> I have much higher expectations for approaches based on yield
Stefan> (i.e. approaches which are inherently safe(r), with the main
Stefan> risk being lack of concurrency rather than race-conditions and
Stefan> corruption).
> Can you expand on this?

I'm not sure what kind of expansion you'd like to see: if you forget
a lock you can get data corruption, whereas if you forget a yield you
may just block a bit more often than desired (but still less often than
in Emacs-23).

Tom> I think right now you can do things like:
Tom> (let ((some-binding))
Tom>   (while (condition)
Tom>   (sit-for 0)))
Tom> ... and expect that some-binding will be visible to the process filters.
Stefan> Yes, that's a code pattern that doesn't interact well with
Stefan> threads.  We'll need to adjust/fix those cases one by one, I
Stefan> think.
> I was thinking of constraining a process to a particular thread by
> default, then letting code set the process-thread to nil to mean that it
> doesn't care where it is run.

That might work, yes.  Good idea.

> But, if you think it is simpler to just fix the code, then I guess we
> could drop this.  My general approach has been to be conservative about
> changes to elisp semantics.

Indeed if we can avoid having to fix the code, it's usually better (the
"usually" is for the case where compatibility has significant adverse
effects on new clean code and the incompatibility is minor).


        Stefan