[Monotone-devel] 3 proposed changes to manifest/changeset format

[Nathaniel Smith]

Just a heads up, and to get some peer-review, on some changes to the
basic manifest/revision format that are probably coming up.  Though
there was some discussion of radical changes related to switching to
a new merge algorithm, at the moment it looks like that will _not_
happen, and the external formats will continue mostly as they are.
Some smaller changes are in the offing, though:

- First-class directory support: basically meaning, put directories
into manifests, and add an "add_dir" operation to changesets
(currently there is "{add,rename,delete}_file", and
"{rename,delete}_dir"; this completes the set).  The big user-visible
consequence of this is that it will make checking in empty directories
possible, which will probably make a lot of people happy (though I'm
still not sure why ;-)).  The really big consequence of this is that
some subtle-but-important cases become much more sane to support;
there is discussion in the bug tracker:
  https://savannah.nongnu.org/bugs/?func=detailitem&item_id=12070
For most people, though -- empty directories.  Good stuff.  This is
almost certainly happening.

How exactly to modify the manifest format is not entirely settled yet.
One possibility is to denote directories by filenames ending in /, and
some sort of placeholder hashes.  Another possibility is to take the
opportunity to switch the manifest format over to using basic_io, like
most other parts of monotone (including, notably, revisions and
changesets).

- Patches on delete_file: Currently, there's an assymmetry in the
change_set format -- add_file "foo" is accompanied by patch "foo" from
[] to [<initial hash>].  delete_file "foo", on the other hand, is not
accompanied by a patch "foo" from [<final hash>] to [].  The reason
for this is that such a hash would be redundant, but I think it's a
decision worth revisiting.

The lack of symmetry alone doesn't bother me that much, really.  What
bothers me more is that because of this, changesets are not inherently
invertible -- invert_change_set has to have access to the pre-state
manifest, exactly so that it can invert delete_file's.  This is a bit
annoying, and certainly slows down change_set inversion (which is
needed by things like diff, annotate, etc.).  (Though I guess it's
possible diff should switch to something more codevilley?  I'm not
sure what that would be, but Richard's rename bug seems to mean that
our current diff algorithm is non-deterministic when it comes to
determining tree rearrangements and file identities.)

It also is in some sense the cause of the one bug we've ever had that
came close to losing history; there's a whole section in UPGRADE on
this bug, that was fixed in 0.17.  When generating the root revision,
monotone left stuff out of the changeset; netsync, then, which looks
almost entirely at changesets, could only see file versions whose
hashes were later mentioned in a changeset, and would silently fail to
fetch some files.  The only files that this applied to were ones that
had been imported, and then immediately deleted without being
modified; if a file was later added, or later edited, or existed in
the head revision, then all of its states were automatically visible
to netsync.  If "delete_file" had included a "patch", then this bug
would never have been able to work at all.

I'm not really sure how great the second argument is, we don't expect
to re-introduce the bug :-).

Anyway, seemed worth raising for consideration.

(graydon, be particularly interested to hear your opinion on this one,
since you haven't seen it before...)

- Attributes in manifests: this is a speculative idea, raised for
discussion -- if we use basic_io for manifests, it becomes pretty
straightforward to include file attributes (like .mt-attrs currently
stores) directly into the manifest, if we want to.  Presumably this
would also involve adding "set_attr" and "delete_attr" type operations
to changesets.

There are a number of possible trade-offs to this.  With cdv-merge, it
should be reasonably straightforward to actually deal with merging
such things.  (They'd be handled exactly the same way as filenames.)
It would still make the merging more complex, though.  It would also
increase conceptual complexity some, I think, to have more sorts of
versioned data, and it would certainly increase UI complexity in some
ways (e.g., we need ways to annotate attributes, signal conflicts in
attribute changes and let the user provide a resolution, etc.).  It
would close the door on making attributes apply to wildcards (possible
use case: set an attribute on "*.pdf" that gave a hint to the merger
how to handle pdfs, e.g., with an external program).  I'm not sure
that this would be a good idea anyway, though.

The motivation behind the current .mt-attrs idea is basically the idea
that, we version files; if you want some data to be versioned
(meaning, old versions stored and retrievable, merging to happen,
etc.), then store that data in a file.  It is still a bit of an
expedient hack, though, and sometimes seems to confuse people.

It's also possible we should just not bite off more than we can chew
right now, with massive merging and storage changes happening already;
assuming we do use a basic_io manifest format, this change would be
backwards compatible.  (If we started including new lines in manifests
and changesets, then all old revisions and manifests would remain
valid, they would just happen to not include any new-style attributes.
If we removed .mt-attrs support entirely, of course, then you would
stop having executable bits automatically set when checking out old
versions; I'm not sure that this is that big a deal, though...)

I'd be interested to hear discussion on this.

Cheers,
-- Nathaniel

-- 
"But in Middle-earth, the distinct accusative case disappeared from
the speech of the Noldor (such things happen when you are busy
fighting Orcs, Balrogs, and Dragons)."