Re: Concurrency, again

[Perry E. Metzger]

On Thu, 13 Oct 2016 08:55:55 +0300 Eli Zaretskii <address@hidden> wrote:
> > Concurrency is a giant tarpit of subtleties. :(  
> 
> And Emacs is another giant tarpit.

Here's another possibility, which I'll admit is even more half baked
(at best) and probably even more excessively ambitious: architect
something entirely new, allow concurrency only for new code, maintain
the old elisp interpreter as single thread only for backward
compatibility, and subsystems that want concurrency can slowly migrate
in to the new part of the system over a period of years.

But, allowing ourselves to ponder the possibility for a minute, this
could solve several problems:

1) There's a perceived desire to move to a new extension language.
2) There's a perceived desire for concurrency, and doing concurrency
well (such that one doesn't end up with data corruption issues here
and there) is hard to retrofit.
3) There's a lot of technical debt in emacs, and a chance to
incrementally move to a new architecture would be a chance to correct
a lot of it.

So the notion (and again, don't take me too seriously, this is very
half baked) would be that one would have a second extension language
with a new design and its own execution engine running alongside the
execution engine for the existing elisp, with relatively minimal
interaction between them to avoid concurrency issues. Over a period of
many years, the codebase would be converted over bit by bit. The new
language should presumably be a lisp that isn't *too* far from elisp,
and a particularly ambitious goal might be to build semi-automated
tools for converting over straightforward code without much human
intervention.

However, all this depends on something I don't know offhand, which is
what the right design for such a new, native-concurrent lisp would be,
or even if a "correct" design is obvious, or even if all of this is
possible. Again, I'm doing the wrong thing here, which is writing a
note long before I've thought about the problem in depth.

I do know that systems that avoid shared memory and do only message
passing seem to be a lot easier to work with than pthreads style
designs where the programmer has to worry that any global state is
potentially in play.

I also know that in the not that distant future processors with
hundreds or thousands of cores are going to be common, so there will
only be more demand with time for the ability to do real concurrency
for performance. Solving the parallel execution issue may be a matter
of importance if (in the future) people are going to do things like
using Emacs as a primary web browser and development environment.

But again, I won't pretend I know how to solve all of this.


Perry
-- 
Perry E. Metzger                address@hidden