concurrency suggestions for Gnus (was: Patch for fields of `struct buffer')

[Ted Zlatanov]

On Sun, 06 Feb 2011 19:34:51 -0700 Tom Tromey <address@hidden> wrote: 

Tom> If we went the "lock anything" route, I would suggest a weak hash table
Tom> for locks, instead of putting the lock into the object.

A bloom filter would guarantee no false negatives, which as you noted is
the vast majority of the cases, requires very little space per element
(about 16 bytes to ensure max 0.1% false positives), and runs in
constant time.  So maybe that's a better first-check data structure in
this particular case with the hash table as the authoritative check.

On Sun, 06 Feb 2011 19:47:51 -0700 Tom Tromey <address@hidden> wrote: 

Tom> My earlier target was gnus, but I think Lars has probably already
Tom> rewritten all the slow bits.  Surely making it fully async can't be far
Tom> behind :-)

Here are some Gnus ideas:

Rebuilding the thread tree in the summary buffer can probably be
parallelized.  For large summaries (over 5000 articles) it can be a long
wait.

The Gnus registry is a way to track messages by message ID.  It needs to
process completely independent lists of data in parallel (classic
map-reduce) all the time, but does it serially right now.  It can
probably add extra baggage to each list to help this tracking, but that
would make the registry less generic and more complicated.  So
parallelizing that search would be nice.

The spam.el library does many spam check/process funcalls.  This can be
slow when the spam processing backend uses an ELisp function and it's
annoyingly complicated when shell commands are used.

Splitting incoming mail can generally be parallelized since each message
is split independently of the others.  This is not true in some cases
but the majority should benefit.

The auth-source.el library could search multiple backends simultaneously
(this refers to a recent rewrite that's currently in a branch in the
Gnus Git repo).  This parallel search is not possible right now.

Ted