Re: Patch for fields of `struct buffer'

[Daniel Colascione]

On 2/1/11 8:19 AM, Stephen J. Turnbull wrote:
> Lennart Borgman writes:
>  > On Tue, Feb 1, 2011 at 1:10 PM, Stephen J. Turnbull <address@hidden> wrote:
>  > > Daniel Colascione writes:
>  > >
>  > >  > The world is increasingly moving toward parallel processing,
>  > >
>  > > Since when does parallel processing == threads?
>  > 
>  > I am a computer illiterate sometimes. Could you give us some hints
>  > about the current status?
> 
> Well, that's actually what I was asking Daniel.  I'm not sure how
> illiterate you think you are, so please try not to be insulted if it's
> all too basic.

That's an excellent overview of the topic. Thanks for writing it, Stephan.

You're absolutely correct in that multiprocessing can also take provide
concurrency the ability to exploit SMP machines. But we've had it for
decades, and it doesn't seem to be used very much to solve some
persistent issues. Sure, flymake, flyspell, slime, M-x compile,
find-grep, and friends allow a lot of work to happen asynchronously. But
it's harder to use multiple processes to parallelize code written in
elisp, especially code that interacts with the user.

I actually prefer the shared-nothing multiprocessing approach in
general, but it's hard to take advantage of that in Emacs today, at
least for elisp. The startup time for a sub-Emacs (at least one that has
also processed user startup code) is high, we have no fork exposed to
elisp for easy data sharing, and there's no simple way to shuffle Lisp
data structures between the Emacs processes, though at least with prin1
and read, the last easier than it'd be in some other systems. You could
argue that bulk processing helpers should be written in something other
than elisp --- and in practice, they are --- but it complicates both
development and distribution.

Have you seen the Python multiprocessing module? It provides a simple,
elegant facilities to coordinate multiple communicating processes ---
things like queues and pipes. A similar approach would work fine for
Emacs as long as we're talking about bulk processing, e.g., parsing lots
of RFC2822 messages, or indexing hairy C++.

But for intricately coordinated work, it'd be a lot harder to allow
these multiple processes to effectively manipulate the same buffer or
communicate with the user: you'd have to either copy the whole thing,
which could be expensive for large buffers and which would have child
operate on stale data; you could provide a protocol for parent and child
to talk about buffers over IPC, but that would introduces complexity,
latency and coordination issues as well as a potentially large number of
context switches; or you could put the buffer in shared memory and
reintroduce some of the problems solved by multiprocessing as well as
run into issues some platforms have with cross-process synchronization
primitives. (ISTR a discussion of these options some time ago.)

We could have two concurrency systems: a cooperative threads system for
tasks inside an Emacs process that work closely with the user and that
share the work of fontifying buffers, and another, high-level facility
for offloading work to another Emacs process. This approach would solve
some problems, but it'd be complex, and I suspect a lot of CPU-intensive
code will nevertheless be run in the main Emacs process because it'd
hard to logically separate buffer-dependent and buffer-independent work
here.

Or, instead of that complexity, we can use shared-everything threads ---
which are inherently preemptive if they take advantage of hardware
concurrency --- and deal with the fallout as best we can; other systems
seem to manage. We could use locks and condition variables, or CSP, or
(my favorite) STM.

signature.asc
Description: OpenPGP digital signature