Re: advice needed for multi-threading patch

[Tom Tromey]

Tom> I'm trying to properly implement per-thread let binding in the presence
Tom> of buffer-local variables.

Stefan> The problem with that is that we first need to figure out what its
Stefan> semantics should be.  And to me the semantics of a let-bound
Stefan> buffer-local variable is very much unclear in the presence of threads
Stefan> (it's already pretty messy even without threads).

Stefan> So I'd rather disallow other threads from accessing this buffer's
Stefan> variable while the let-binding is active.

I thought I'd post a status report on the multi-threading patch.

First, thanks to Giuseppe, it is all hosted on Gitorious:

    http://gitorious.org/emacs-mt

'mt' is the branch I've been working on, he's done some additional fixes
and changes on g-exp.  (I am behind on pulling from him... and we don't
agree about everything, hence two branches :-)

What works:

* You can start a new thread with run-in-thread.  The exact use differs
  between the branches.  Threads are cooperative, you can yield with an
  explicit call to yield.

* The GC handles multiple threads.

* Many, but not all, of the 9 kinds of variables that Emacs implements
  work properly with threads.  In particular, working ones are
  defvaralias, objfwd, and ordinary.  These all should do the right
  thing in all cases.  Buffer-local and buffer objfwd mostly work,
  because we do buffer locking.  (However, I think make-local-variable
  doesn't work properly with let-binding and multiple threads.)  Not
  working on intfwd, boolfwd, keyboard-local and frame-local.

* Buffer locking.


There are also a number of problem areas.

* Buffer locking is actually quite a pain.

  If you have a background thread locking some buffer, you can't switch
  to that buffer.  Emacs will just pause.  I don't think we allow
  interrupting a lock acquisition (clearly a bug).

* Giuseppe implemented minibuffer exclusion.  But really the keyboard
  should only be available to one thread at a time.

* I think there are some V* variables that probably should always be
  per-thread.  My notes say Vsignaling_function, but I know I've run
  into other ones.  (What is noteworthy about this is that it implies
  that a big audit of the C code is necessary..)

* I/O is also an unsolved problem.  Right now it is easy to make emacs
  appear to hang by running a process filter in one thread while a
  different thread owns the buffer lock.  (Probably, if a thread owns a
  buffer lock, then only that thread should be able to run associated
  process filters.)

I think Giuseppe probably knows of more problems.

Also, there are tons more problems if you want preemptive
multithreading... redisplay probably needs work, regex is not reentrant,
the GC needs a bit more work, you need lisp-level locks, etc.


Sometimes I wonder whether it would be better to just fork a second
emacs and communicate with it using pipes + princ + read.  This uses
more memory but the implementation would be a lot simpler and (I think)
it would suffice for the one case that seems most important: Gnus.


Anyway, give it a try.  We welcome your advice and input.


Finally, we've been doing all our discussions privately on irc.  I'd
prefer we discuss everything in the open.  Is it ok to use emacs-devel
for this?  If not we'll set up a new mailing list somewhere.

Tom