Re: [VM] Maildir support

[Uday Reddy]

On 6/27/2011 11:30 PM, Kurt Hackenberg wrote:

> That model -- a folder is a file -- doesn't fit an IMAP folder, or a
> maildir folder.  I can think of ways to make it work...  How does an
> IMAP folder work?

Perhaps, I overstated my case earlier.  There is definitely a data 
structure *associated* with a VM folder buffer.  The data structure is a 
list of message descriptors, one for each message in the folder.  The 
message descriptor stores all kinds of data about the message including, 
in our case, the external location of the message if it is externally 
stored.  The "Message Internals" section in the current VM manual 
documents what is contained in the message descriptors.

Normal IMAP folders are just like ordinary VM folders, except that their 
message descriptors store the location information on the IMAP server as 
well.  This includes the unique id (UID) of the message on the server. 
When VM synchronizes with the IMAP server, it retrieves all the message 
UIDs on the server and the message attributes (like 'deleted', 'flagged' 
etc.).  Then it takes the actions necessary to bring the two folders 
into sync.  (If a message exists on the server, but not internally, it 
is expunged on the server.  If a message exists internally, but not on 
the server, then it is expunged locally.  So on.)

VM's IMAP folders allow "external messages" (formerly called 
"headers-only" messages).  These are messages in the VM folder with 
their message body missing.  However, some kind of a message skeleton 
would still be present in the VM folder.  The message descriptors record 
the fact that the message body is missing.  When VM runs into one of 
these messages, it goes to the server, fetches the message body, inserts 
it in the missing gap, and proceeds as usual.  There is a limit to how 
many of such transient messages VM keeps, say 10.  If the limit is about 
to be exceeded, it erases one of the message bodies it had fetched 
earlier to make room for the new one.  When the IMAP folder is saved to 
the disk, all these transient message bodies are erased.

The same method can now apply to all other kinds of external messages. 
For maildir, the sync operation would read the maildir directory to find 
out what messages exist in the external storage and bring the VM folder 
into sync.

For NNTP news servers, again, the principle would be the same, except 
that there is no capability (and no need) to expunge messages on the 
server.  Presumably RSS feeds can be handled in a similar way too.

All this seems quite elementary, once it is worked out.  But Kyle Jones 
seems to have believed that VM was fundamentally an in-memory mail 
client that could not deal with external messages.  Rob F realized that 
it was easy enough to deal with them, but his idea was to bring external 
message bodies into VM's Presentation buffers rather than Folder 
buffers.  I started with that idea:

   http://www.emacswiki.org/emacs/ViewMailHeadersOnly

but found that it didn't quite work.  It was also quite clumsy because 
moving around in the Folder caused a lot of repeated fetching from the 
server and MIME decoding all that work.

Fetching message bodies directly into the Folder was a bit scary at 
first because I needed to have global knowledge of everything VM was 
doing with folder buffers.  It took me a while to re-engineer VM to 
allow external messages.  But, once this is done, all other kinds of 
external messages can easily fit in.

I would guess all other mail clients that deal with external messages do 
pretty much the same thing, e.g., Thunderbird.  But they would have 
started with external messages forming a fundamental part of the 
requirements.  So, their initial designs would have taken care of all 
these issues.

The main limitation of external messages is that they don't allow 
searching inside message bodies.  But it is possible to get around the 
problem for IMAP and maildir.  IMAP servers have queries for searching, 
which can be used to "run the search on server" (as Thunderbird calls 
it).  For maildir, there are external applications like Mairix that can 
be used.  All of these are on the To-do list.

Cheers,
Uday