Re: [Monotone-devel] A Two-Fold Proposal: On Formats And Front-Ends

[Larry Hastings]

Well, the proposal has been up for six hours, and no threats of bodily harm yet.  So that's something--guess y'all are friendly after all.  My thoughts on the replies so far follow.


Joel (address@hidden) wrote:

I notice that JSON doesn't support circular references in serialized object trees. Is this going to be a problem?

I'm not really qualified to answer, but... I doubt it.  The current dazzling myriad of text formats doesn't support any explicit circular references either, and yet monotone works fine.  Whatever circular references already exist are implicit in the data being stored.


Julio M. Merino Vidal (address@hidden) wrote:

What about YAML (http://www.yaml.org/)? It could be useful too.

That's a new one on me.  Looks okay at first glance.  It's way more complicated than JSON, but then that's because it supports a much richer set of types (sets, binary values stored in base64, a reference scheme).

Also, they don't seem to provide a reference implementation for C, just for several scripting languages.  There does appear to be a C implementation here:
    http://tpe.freepan.org/repos/oren/yaml-parser/
but it hasn't been updated for over a year.

I think I prefer JSON.  I don't think monotone would benefit greatly from the larger set of types; JSON has a "right-sized" feel to me.  But if monotone switched to YAML, I wouldn't complain, I'd roll up my sleeves and get to work.


Hendrik Boom (address@hidden) wrote:

Would (multiple) front-end(s) and a shared engine library do?

Perhaps, perhaps.  My goal is to write a front-end in Python.  For that, an executable where I talk over pipes is preferable to a shared library.  But if one of the front-ends provided exactly that interface, I wouldn't mind.

One of the reasons I proposed splitting it into two executables was to erect a "chinese wall" inside monotone, to keep it honest.  If every operation ever run from the command-line went through this pipe interface, then there would be no possible way to circumvent it.  If the interface is a shared library, someone could say "Doing operation x with these ding-dang pipes is a pain in the keister, so I'm going to make an end-run around the pipes and make up a special API call that makes this way easier for me."  If the only interface is the pipe, they wouldn't have that option, and they'd have to "suck it up".

[XML] fails to distinguish a type from a field-selector. [...]
ASN.1 makes the same fatal mistake. [...]
JSon also fails to distinguish between types and field-selectors.

I'm afraid I don't understand the distinction you're drawing.  Can you elaborate?

And is this something that you know monotone would need?  Right now I doubt monotone's various output formats make this distinction either.


Alex Queiroz (address@hidden) wrote:

What a lot of people forget is that Monotone has already a parser for a very powerful data description language: Lua. Although Lua is a modern language with co-routines, tail calls, anonymous functions, lexical scoping etc. it was first designed as a data description language. If the Lua syntax was used thoroughly in Monotone, that'd be zero effort in writing or reusing another parser.

Zero effort for monotone, but great heaping handfuls of effort for anyone else.  If the first step in writing your own monotone front-end was "call the Lua parser", it would make writing front-ends a major headache.  Particularly for people not writing their front-end in C.  (There are, understandably I think, no Lua bindings for Python, Perl, OCaml, and the like.)

Also that that wouldn't help with the output data.  monotone would still need to write an output formatter.  Though I acknowledge that this piece would be pretty easy.


Marcel van der Boom (address@hidden) wrote:

implement a --format switch which takes something like a printf() format string/file, where you can deviate from the standard output format.

I agree, that would make writing front-ends using the existing monotone executable simpler.  But it wouldn't solve the "tower of babel" problem with respect to file formats.

Also, it wouldn't allow any real flexibility.  For instance, what would you do with monotone automate graph, where you list multiple ids per line?  How would you format monotone automate inventory?  Or monotone status?  How would this handle error conditions?

The approach you propose would solve 80% of the problem.  I'd rather solve 100% of the problem, even if the solution is more complex.  As Mr. Einstein once said, "everything should be made as simple as possible, but no simpler."

Having to have a separate daemon to set up would not be very welcomed by me.

Right now this is all blue-sky initial planning, so I get to play Santa Claus and say things like "well, of course you wouldn't be forced to use monotone in daemon-mode."  Anyway, I doubt we'd want to change to that being required; as you say, having monotone being this standalone non-daemon thing is very pleasant.


Bruce Stephens (address@hidden) wrote:

So you want to design a shared library API, instead?

I'd guess it's a little easier to engineer extensibility in a protocol than a library, although both can be done to some extent.

If the shared library API consisted of only two calls: "open two pipes" and "let monotone chew on the data waiting on its incoming pipe", and then the pipes carried JSON-like data as per my original posting, then obviously this would satisfy me.  Indeed, it'd be exactly as extensible as the approach I proposed.  One front-end could be the conventional monotone user-friendly front end, one could simply expose the pipes to the outside world.  m7 would eschew the former and use the latter.

I suppose one could even take this approach without splitting monotone into two executables.  If internally the "chinese wall" was observed everywhere, and the two-pipe method was used for everything, and the command-line allowed external applications to get at the two pipes, then that would be okay too.  (We could still have the "take this blob of JSON and turn it into a pretty localized message for the user" interface on this monolithic executable, too.)  Though again I fear people digging under the "chinese wall".

I guess a precondition for this kind of thing is proper locking of repositories.

Certainly!  I was expecting that the pipe-based API would have explicit locking semantics, as well as an implicit lock on every command which writes to the database.  I gather monotone actually has relatively-little locking, which I find surprising.

I imagine that those operations which monotone now offers are almost all the atomic operations that are needed, and that m7 just wants a few more, and maybe they're easy to provide (because they're things like committing and creating a new tag, which are database operations)?

That's how I see it.  The pipe-based API would be slightly lower-level than the current API, but not much.  Add explicit locking, break up the querying commands so one can construct simple queries (a higher-level approach to "select TAG from REVISIONS where NAME = VALUE" stuff, to abstract away the schema), and we'd be most of the way there.

We might want a simple request/response protocol here, in case monotone needs more information about something mentioned in a request, or to break up requests into smaller pieces, or to allow front-ends to be dumber.  As a possible example:

c -> s: Request 1: Please commit the current changeset.
c <- s: Request 1: Please send me MT/work
c -> s: Response to request 1: <<contents of MT/work>> (changed: a.txt b.txt)
c <- s: Request 2: Please send me a.txt
c -> s: Response to request 2: <<contents of a.txt>>
c <- s: Request 3: Please send me b.txt
c -> s: Response to request 3: <<contents of b.txt>>
c <- s: Response to request 1: Commit complete.  Here is your new MT/work <<contents of new blank MT/work>>

The obvious approach would be to make the file contents an explicit part of the request.  However, that means building smarts into the front-end (like understanding what files have changed in a change set so it can send them).  Also, if there was a problem in the request, monotone wouldn't be able to reply to the request and yield an error until the request was fully-formed, delaying reporting the error until you ship that 400MB Photoshop document over the pipe.

Just a thought for now.


Cheers,


larry