Re: [Nmh-workers] Locking (specifically, sequences)

[Ken Hornstein]

>Spitballing for a moment: I wonder if we can detect the case where the
>sequence file changes out from under an operation.  If so, it might be
>possible to save the original data, compute a delta of some sort, and
>use that to repair the sequence, or punt to the user?

Maybe I didn't explain it that well ... that's kinda what I was talking
about.

Specifically, there are two main ways I'm talking about dealing with this:

- The "simple" case, where the sequence file is write-locked the entire time,
  from folder read until modification.  It should be obvious that this
  shouldn't present any problems with any other programs (unless you're
  using something like Claws Mail, of course).  It should be noted that
  the window here is actually small in terms of time; for example, "show"
  does all of it's sequence manipulation before it actually displays
  the message.

- The "complex" case, where the sequence file could potentially be changed
  from the time that it's first read until the sequence file needs to be
  manipulated.  "inc" is an example; "pick" is another.  I believe that
  re-reading the sequence file right before you need to operate on it
  should solve that problem.  Although there might be some tricky technical
  issues there in terms of nmh internals, I don't think they're insurmountable.
  
There are, I believe, a few extra corner cases that can't easily be dealt
with give the current code structure.  The basic issue there is if you
add a new message to a folder, the long-running command wouldn't necessarily
know the new message existed and thus when it wrote the new sequence
file out it wouldn't write out any sequences that these new messages
are in.  So, for example, if you used rcvstore to receive messages into
your inbox and a message came in during a long-running "pick" command,
if pick wrote out a new sequence then it would probably not have the
new message as part of the unseen sequence.  Things like that.  Note
this only applies to "long-running" commands; as as I see it we only have
two of those in this category, inc and pick, and adding new messages to
a folder that you're currently inc'ing into is already not safe (messages
could get overwritten).  pick is the only candidate on the table for
this issue.  It also occurs to me that reading the whole folder again
after a pick is done searching would solve that issue ... and looking at
the code, that doesn't even look that hard.

>I used to see this problem (rarely) with a periodic fetchmail+slocal
>setup, and 'folder -pack' was almost always enough to make the world
>right.

I'm curious; what would happen?  The sequence file would get corrupted?
I don't think that could happen nowadays.

>On the other end, things like rsync, hg, and git have advanced
>the state of the art in version-skew management a fair bit since those
>days.

Well, I'm not sure that their technology is applicable.  In my view
git & hg provide advanced management of a set of changes, but when doing
merges you're still doing a classic three-way merge that punts problems
to a developer.  I don't think we want that for nmh (nor do I think
it is necessary).

--Ken