Multithreading, again and again

[Dmitry Antipov]

Hello all,

this is an attempt to refresh the most important ideas everyone has about
threads support in Emacs. Thanks in advance to all who spent their time
thinking on answers. If someone thinks that I miss something important,
notices and references to previous discussions around this topic are highly
appreciated.

0. What we have, and where we go.
   What's the status of git://gitorious.org/emacs-mt/emacs-mt.git and
   http://bzr.savannah.gnu.org/r/emacs/concurrency? Whether there are
   plans for further development?

1. To thread or not to thread?
   Is there a consensus whether it's worth doing at all? If I don't miss
   something important from previous discussions (referenced from
   http://www.emacswiki.org/emacs/NoThreading, what a title), everyone
   agrees that some kind of threading is useful - but maybe someone has
   the votum separatum?

2. Hide them all!
   Should the threads be visible (may be created and managed) from Lisp,
   or they can just do some work orthogonal to it? If latter, which tasks may
   be implicitly threaded? Redisplay (one thread per frame)? Buffer/socket
   I/O? GC? Is it feasible to implement full implicit concurrency? For the
   rough example, with the Lisp engine as the pool of threads, where (eval
   FORM) just queues the FORM and then ask the pool to find a free thread
   and evaluate it?

3. Too big, too global.
   How to split/share/synchronize huge global state between threads?
   What should be thread-local? (I feel (current-buffer) should be
   global and max-lisp-eval-depth should be thread-local, but I suspect
   that there are much more controversial cases). Is it reasonable to
   have become-thread-local-on-write globals?

4. Say "thread"!
   What should be done in C to implement thread management in Lisp?
   I feel basically

     (let ((th (make-thread FUNCTION ARGS))) ...)

   should create thread object TH which represents running thread executing
   Lisp function FUNCTION with ARGS (like funcall, but within separate thread),
   and such thread object may be examined with stuff like (thread-running-p TH),
   canceled with (thread-cancel TH), asked for FUNCTION's return value with
   (thread-return TH), and so. But is this nothing more than adding some
   builtins? Will it require some more substantial language changes? What
   about old controversial topic on lexical vs. dynamic binding?

5. You and I.
   What synchronization primitives should be implemented? Atomic variables?
   Mutexes/semaphores? Barriers? Higher-level not-so-primitives like
   thread pools or multithreaded queues? Should some basic operations,
   like setq, be always atomic? Is it reasonable/feasible/useful to have
   special form like (with-atomic BODY...), where all BODY forms are evaluated
   atomically with respect to all other threads?

6. Under the cover.
   What other low-level changes are required for thread support?
   Basic structure of Lisp objects? Multiple stacks scanning for GC?
   Per-thread heaps for small objects?

7. Look around.
   Has someone an experience with development of multithreaded code for
   commercial Common Lisp implementations, most probably LispWorks and/or
   Allegro CL? If yes, can someone briefly explain pros and cons of their
   approach to multithreading?

Dmitry