Re: The sad state of locking -> it almost works

[Alain Magloire]

> 
> On Fri, Apr 12, 2002 at 09:30:17PM -0400, Alain Magloire wrote:
> > > 
> > > FWIW, what I've been up to is more-or-less re-implementing something
> > > along the lines of the API of the original libmailbox.  There are a
> > > couple of reasons I've been doing this.  The foremost reason is purely
> > > as an academic exercise.
> [SNIP]
> > > Anyway, you guys are doing some really cool things with mailutils, and
> > > I like the idea of url access especially.  But I'd rather see that
> > > implemented in a different layer, on top of the physical access layer.
> > > It seems to me mailbox kinda tries to handle everything all at once,
> > > and I'd rather break everything down into layers -- physical layer,
> > > application layer (like dealing with MIME and e-mail addresses and
> > > lions and tigers and)...  
> > 
> > Good points, we will do that: two layers, the first one dealing with the
> > mailbox format nothing fancy i.e. get the job efficiently,
> > the second layer: the framework to do all that fancy things.
> 
> I was originally thinking of 3 layers -- the third to deal with
> accessing e-mail over a network.  Though now that I think more on it,
> I don't think this is quite so much a "layer" as I think the code for
> this module (or set of modules) is largely seperate and distinct,
> sitting at the same level as the physical mailbox access stuff.  I
> think a diagram might look something like this:
> 
>               --------------------    -------------------
>               |                  |    |                 |
>   ACCESS      | physical mailbox |    | network mailbox |
>   LAYER       |  access module   |    |  access module  |
>               |                  |    |                 |
>               --------------------    -------------------
>                            \            /
>                             \          /
>                              \        /
>                         ---------------------
>                         |                   |
>   APPLICATION           |    application    |
>   LAYER                 | data manipulation |
>                         |     module(s)     |
>                         |                   |
>                         ---------------------
> 
> So I guess not so much three layers as two, with two seperate modules
> (well, groups of modules... more on that in a moment) in the access
> layer, and one or more module in the applicaton layer.  I might also
> do well to rename physical mailbox access as on-disk mailbox access...
> I think both that and network access qualify as physical access,
> though of a very different nature.  

Actually three, a common API; mailbox_t and friends
which allows you to talk transparently to the "Access Layer".  The
common glue to chose a mailbox in particular is the URL, pop://, imap:// etc ..

> Having not ever really dealt with writing an e-mail application, I
> haven't really given much thought to what kinds of modules might be
> needed in that second layer.  I guess functions for processing MIME
> components of a message would certainly need to be one of those
> modules, and I suspect you'd need functions for parsing e-mail
> addresses...  prolly some other stuff.

Mime, encoding, decoding, search, batch processing, notifications, calendars 
etc ...
can be tied on top.  In the case of IMAP it is a little fuzzy those operations
are provided on the server side.

> Ok, so the "modules" shown above in the access layer are not really
> individual modules, but sets of modules.  For example, the physical
> mailbox access "module" would contain modules for accessing mbox, mbx,
> mmdf, maildir, mh, etc.  The network mailbox access "module" would
> contain modules to handle POP3 connections, IMAP connections, SMTP
> connections, etc.  They're not modules but groups of modules.

Sure.

> Ideally, the code for all of these modules is written modularly enough
> that no module in the access layer need know anything about the code
> in any of the other modules.

Yes this is what we are missing in our current offering.  My mailbox
is actually full of comments on this.

> The modules in the application layer
> need know /only the exported API from the access layer modules/. 

Yes, for example when Sergey was doing MH some of the features of MH
did not have any correspondance in the other formats and we kind of loose it
by forcing every body to same mold.  If one wants to use the "Application"
layer then the choice is a trade off for other advantages.

> With
> one exception, that is: they all obviously need to know the definition
> of a mailbox_t structure.

Only in the Application Layer.

> servers will each do both functions, I think of network access as
> being mostly a function of the client applications, whereas on-disk
> access is more a function of the server applications.  All I'm really
> saying here is that to a server, the on-disk view of a mailbox is more
> important, whereas to a client, the on-disk view of a mailbox is
> becoming decreasingly important, as POP and especially IMAP become
> more and more widely adopted.  I don't know if that makes any sense to

I think POP and IMAP are widely adopted.

> anyone but me, but if not that's ok.  If it doesn't make sense to
> you, just ignore this whole paragraph.  :) ]

8-) In our case the line is blury, a server can become a client.
For example an IMAP server can become a proxy to contact another server
whether POP are IMAP.

> > As you are pointing out one should be able to use say pop3 without
> > having to go through the entire framework layer.
> 
> Yeah, I agree that makes sense.

OK.

> > So if you are working on one of the formats: POP3, Mailbox, Maildir,
> > IMAP4, MH etc ..  it probably can be reuse once we move on to the
> > second stage.
> 
> Well, right now I'm concentrating on mbox, because I've been somewhat
> fascinated with performance issues surrounding that format ever since
> the first time I started using UW-IMAP.  I'm experimenting with a few
> things: As I've already mentioned several times, I'm playing with MMIO
> vs. file I/O.  I'm  also screwing around with ways to eliminate temp
> file usage by re-writing the mbox file *in place* and only re-writing
> that which is essential to be re-written.  My current theory is that
> performance can be enhanced significantly by doing this, rather than
> by re-writing the whole mailbox to a temp file and moving it back to 
> where it goes...  I've also got a similar idea about how to modify
> message status (and possibly other frequently modified headers, like
> X-UID) without re-writing ANY of the mailbox, other than the status of
> the given message itself, which would obviously increase the
> performance of such an operation tremendously.  If you like, I can go
> into more detail about my theories on this.  But right now, I need to
> eat some lunch...  ;-)

We need code 8-).  Must of the ideas are clear.
But right now I must admit having no time, I'm on a very tight
schedule to build a new product that must be out in June and even
weekends are used.  But after that ...

But if you find time to right code, for example the module of
the access layers you are referring too, I'm sure it can be reeuse
and reintegrated.

> Once I get done monkeying around with all that, I'd probably be
> interested in tackling maildir, as I think it also represents an
> interesting academic challenge (though I gather Dan J. Bernstien is a
> genius, and simply reading his code would be sufficiently
> instructional).  I think I've also mentioned to you off list my

hum ... I hear that a lot.

> interest in IMAP.  However, I start a new job tomorrow (yay! a
> paycheck!) so my time to mess with all this will be much more limited
> now.  And I can't really pull 25 hr. coding stints like I did the
> other day... ;-)  So I'll probably be messing with mbox for about
> another month or so.

Cool.

Later.

> > You are probably a little harsh on our first attempt, but it's not
> > that bad and we learn a lot from it and most of the code will
> > probably be backported with some care not too make the same mistakes
> > 8-)
> 
> Oh well hey, it was not my intent to be at /all/ harsh.  I just find
> the current incarnation of mailbox doesn't quite suit my needs (and
> also that with what I will learn in the process, I will personally
> benefit greatly from trying to implement most of this stuff from
> scratch myself).  Like I said, I think you guys are doing a lot of
> interesting and useful things with the library.  I would also comment
> that I have not looked all that closely at very much of the existing
> code (though some of it), so I don't really have the insight with
> which to fairly make harsh criticism! ;-)  
> 
> Comments on all of the above are quite welcome, and I take criticism
> quite well (even harsh criticism, usually :) so feel free...
>