[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Monotone-devel] Testresult
From: |
Timothy Brownawell |
Subject: |
Re: [Monotone-devel] Testresult |
Date: |
Sat, 09 Jan 2010 23:42:04 -0600 |
User-agent: |
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091204 Thunderbird/3.0 |
On 1/5/2010 5:25 PM, Judson Lester wrote:
On Sat, Dec 26, 2009 at 2:30 PM, Timothy Brownawell <address@hidden
<mailto:address@hidden>> wrote:
On 11/24/2009 1:48 PM, Judson Lester wrote:
Following the bisect thread brought some questions about
testresult back
to mind. This was one of the features of monotone that drew me
to it,
back around 0.14, and I've never really gotten around to really
implementing testresult in my code projects.
Is there a best practice on using the mtn testresult command?
No. In fact I'm not sure that it's even used much, probably related
to what you mentioned below. I think I know a better (more flexible)
approach, but it would need new hooks to be added.
<snipped>
What would probably work would be a separate branch containing
(only?) a file with the list of (test-id, pass/fail) pairs;
revisions in this branch could be given an extra cert (because certs
are indexable, and this is something you'd use for lookups)
indicating which revision the contents refer to, and could maybe
have an extra file with information on what version of the testsuite
was used (for external testsuites).
Doing it this way would require a new "accept_update(old_revision,
new_revision)" hook, and helper functions to allow that hook to look
up particular revisions (use 'automate select') and then read the
contents of those revisions (use 'automate get_file_of'). (We
already have an 'mtn_automate' function that would allow these,
except that it will E() on its reentrancy check if called this way.)
Does this seem like a reasonable approach?
Re-reading this, I started wondering if this might not be related to
policy branches. I am certainly going to reveal my novice-level
understanding of deep version control, and possibly a complete
misunderstanding of policy branches. First of all, I know this was a
hot topic in monotone about 3 years ago, but did they go anywhere?
I have a branch I've been working on on-and-off for maybe a year,
currently I need to finish updating it after the keys-by-hash changes.
It took me a little while to grasp the necessity of a second branch,
although it's obvious on reflection. In order to generate the
testresults, you need to have the actual revision - you could only do
that after the user issues a commit and before it's committed, and even
then, that's a sketchy proposition.
So it makes sense, then, to put the results in a cert that can be added
later - an actor can check out the revision, run tests and then sign
certs on the revision. Except for the complexity of the per-testresult,
and the data bloat you alluded to.
What I started to think about then, was to commit the testresults in a
clean format (maybe something parse_basic_io ready) into a branchless
revision, and then sign a cert on the revision being tested with the
rev_id of the testresult revision as the value.
I'd been thinking to have the cert be on the testresult revision, and
point to the revision that was tested. Doing it the other way gives an
extra benefit that even if you don't pull the testresult revisions you
know that they're there, nice. :)
It seems like you could build a selector for the "associated branch"
pretty easily, and be able to check it out and examine the contents of
the testresult.
I think there'd be a hurdle with netsync, since I think (but don't know)
if I'm syncing with a pattern, revisions that don't match the pattern
would be excluded - so the associated revisions wouldn't make it over
the wire.
This is why the testresult revisions *do* need branch certs - so you can
ask for them during netsync. ;)
There's also questions about the DAG of the associated branch...
Hmm... I guess the simplest thing would be to have the test bot(s) just
write out their results and commit in a loop, and not care whether there
were any other bots. So you'd end up with a linear chain of revisions
for each bot, and with multiple bots you'd get multiple heads. I don't
think the DAG would matter that much, since it really is a bunch of
independent revisions that just happen to be committed as a DAG to take
advantage of the compression.
From there I begin to wonder: does that look at all like policy
branches as they've been imagined? All I've read is the Graydon
proposal, that implies entirely separate branches - sort of filling in
under (e.g.) net.venge.monotone.mtn with net, net.venge and
net.venge.monotone with update policy.
Here, each testresult revision contains info about a specific primary
revision. With policy branches, it's the branch (so, the current head
revision) that contains info defining the primary branch (what
committers to trust, etc).
Think of revisions of a branch as files in a directory... a policy
branch is the ACL on the directory and maybe the directory name;
testresult revisions would be extended attributes on the files (just
like certs would be).