[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Gnu-arch-users] Arch Roadmap Draft (the anticipated part 3)
From: |
Tom Lord |
Subject: |
[Gnu-arch-users] Arch Roadmap Draft (the anticipated part 3) |
Date: |
Thu, 1 Jul 2004 13:58:46 -0700 (PDT) |
Arch Project Roadmap
* Background
For a few weeks, some community members have quietly pressured me to
make a roadmap for future arch development and for some of that time
I've promised to do so and started work on it.
As gnu-arch-users readers are aware, I started stating the plans in a
"bottom up" way: listing a large number of feature plans and then some
low-level details of furth. This makes a lot of sense to _me_, at
least, because before I can be comfortable summing up the "big
picture" I have to be confident that the big picture sums up a set of
definately doable, real ideas rather than being a hopelessly vague set
of "goals" like "improve the user interface" or "simplify
deployment". In some sense, the "big picture" is just a set of
smaller steps, each of which can clearly be done, and which relate
to one another in synergistic ways.
While working on the plans and on enough of the code to convince
myself the plans could be done, I paused merging and release making.
It seems to me that between 1.2 and 1.2.1 we have stable and
reasonable releases out there for people to use. It seems to me that
while the set of pending submissions has some nice things in it, none
of them are especially time-critical and, meanwhile, I'll have a better
idea of which and how to merge those once I have the long-term plan in
place.
The combination of those two actions, posting lower level feature
plans and pausing merging and release making, drew an _interesting_
response from the user community. Basically, it pissed a number of
people off and they started complaining at me.
It's not really desirable to piss people off but sometimes, as in this
case, it couldn't really be avoided while at the same time doing my
actual job of designing and building out arch.
However, while it wasn't my intent, it turns out that it was at least
a little bit _useful_ to piss people off because among their
complaints are some insights into what priorities will, in the longer
term, keep the user and contributor communities happy.
Here's a roadmap for future Arch development. I suppose that this
counts as a "draft": revisions may or may not be needed depending on
the feedback from various quarters.
* Overview
The roadmap comes in two parts:
~ Issues to Address
~ Plans of Action
In outline form, these expand to:
~ Issues to Address
~~ user complaints
~~ contributor complaints
~~ maintainer complaints
~ Plans of Action
~~ Strategic Overview
~~ Project Process Changes/Stable Branch
~~ Feature Plans and Plans of Attack
~~ Contingencies
~~ Release Map
* Issues to Address
The roadmap is intended to address _real_ needs, concerns, and
(most importantly) opportunities. It would easy to just
"make up" a direction to go, but it seems to be far too late
in the lifecycle of the project to do that. If the real needs
aren't met, there are these days enough people who will complain
about that the plan won't fly.
I've classified each of the issues to address by an (idealized)
classification of who is most directly concerned with
the issue: users, contributors, or me, the maintainer.
The category "Users" represents issues that arise in day
to day _use_ of arch.
"Contributors" represents issues that pertain to the patch-flow
and release-making within the GNU Arch project from the
perspective of people other than me who contribute code.
"Maintainer" issues are those that express my concerns about
the design of Arch, based on my observations of how it is being
used and my knowledge and beliefs about what is possible.
** User Complaints
A simple, short grocery list, in no particular order:
1. Revlibs glitch in some NFS environments in a way that
requires them to be rebuilt frequently. (This is fixed
in mainline but not yet in a release.)
2. There are too many commands.
3. Some of the error messages are still bogus.
4. Tagging methods are hard to understand and use.
5. Performance on large trees is not up to snuff yet
for some operations.
6. It's hard, in some environments, to properly configure
a remote archive which can be written by more than
one person.
7. There isn't a bundled graphical merge tool.
8. There isn't a bundled GUI.
9. There isn't a built-in way to prune patch logs and
keep them at a managable size.
10. In some situations, it is hard to know how best to
manage archive cached revisions and the like.
11. The documentation needs some improvement.
** Contributor Complaints
1. Since April, Tom has been much too slow (essentially halted) at
processing submissions.
** Maintainer Complaints
1. Arch is underutilized. No, I don't mean that there
aren't yet enough users (which sure, is likely to be true
for a very long time). Rather, the existing users (including
me) aren't making as good a use of arch as we could be.
There's a fairly large gap there, I think.
One example is the very ad hoc way that we exchange
patches, even when we're using arch for that exchange.
It would currently be impossible (just because we're
sloppy and inconsistent about patch submission) to
do something obvious and useful like set up a browser
that can show all of the changes that are in the
Bug Goo queue.
Another example is the fairly ad hoc (and inconsistent, over
time) way that I tag releases. If you want to go back
and get various historic releases out of the archive, you
have to do some poking around to figure out how the contents
of each release were recorded.
2. Too many conflicting demands on my time suggest the need for
some process improvements.
There's been grousing, lately, about my backlog of merges.
The actual arch work of processing these changes can be pretty
fast but I'm hampered by my slow modem. I have to wait for
changes to be downloaded locally, I have to push my merges
back upstream. I must cut releases on my local machine yet
the subsequent uploads, announcement posting, mirror updating,
and so forth takes a substantial part of a work day for each
release: a day mostly spent just waiting and checking
periodically that modem lights are still blinking. As
streamlined as even something like FreshMeat is it's _still_
a disruptive pain-in-the-ass to go through all that for each
release.
Big deal, right? Except that conflicting with the need for
me to do that busy work is the need for me to _think_ about
where to take arch from here and to actually do that work.
And, there's demands on my time that are not _directly_ arch
related such as hackerlab and pika hacking.
3. The market for arch-related labor exists, but there aren't
enough customers. Therefore, there is a danger of corruption
of the project.
Of course we want the market to grow in dollary terms, that's
obvious.
What may be less obvious is that currently, there are only a
small number of employers who pay for arch work. Consequently,
there is a lack of diversity of interests in the marketplace and,
not to be chicken little, but if we aren't careful, the interests
of those few employers might come to dominate the future of arch,
at the expense of our larger community interests.
We need to make arch easier to deploy in order to attract
new commercial interests to it. We need to make sure that no
one or two employers is in a position to "take over" arch.
4. I share some of the user complaints or complaints close them.
Specifically:
User complaint: There are too many commands.
Tom's take: There aren't enough commands. What's missing are
a few higher-level commands that capture the most common and
most recommended use patterns of arch. Once those are in
place, most users can focus on just those few commands and
ignore the rest. An improved help system can focus attention
on just those few commands.
In other words: Arch doesn't have too many commands but,
currently, average users have to use too many of the commands
to do their work. Consequently, they tend to be a little
daunted at the number of commands they have to learn to
get started, and the help system listing of commands has
to list an intimidating number of commands.
User complaint: Some of the error messages are still bogus.
Tom's Take: A valid complaint that continues to be answered
by incremental improvements.
User complaint: Tagging methods are hard to understand and use.
Tom's Take: Again, some higher-level help is probably needed
to capture common patterns.
User complaint: Performance on large trees is not up to snuff yet
for some operations.
Tom's Take: The largest improvements needed in this area are:
summary deltas; fast, general, selective commit; and improved
ease-of-use for (a generalization of) configs.
User complaint: There isn't a bundled graphical merge tool.
User complaint: There isn't a bundled GUI.
Tom's Take: The bottlenecks here are my programming environment
at home and my time. The roadmap aims to improve the use of my
time, but the programming environment is a harder problem. In
particular, I use a somewhat (by today's standards) slow
and low-memory machine with limited disk space running an old
operating system and X11 release. I simply do not have the
facilities on hand to _build_ most of the candidate merge tools
let alone contemplate integrating them with arch. Even if
someone else were to do that work, I lack the facilities for
examining and testing their results!
User complaint: It's hard, in some environments, to properly
configure a remote archive which can be written
by more than one person.
User complaint: There isn't a built-in way to prune patch logs
and keep them at a managable size.
User complaint: In some situations, it is hard to know how best
to manage archive cached revisions and the like.
Tom's Take: These are all aspects of the more general problem
that it isn't easy enough yet to deploy arch in
some slightly complex situations. I think that
the solution is the addition of some additional
high-level commands (and support for those)
to further automate some "best practices" for
deployment.
User complaint: The documentation needs some improvement.
Tom's Take: Yes, it does.
* Plans of Action
So, given all those complaints, what's to do?
** Strategic Overview
Two broad themes emerge from the lists of complaints:
1. The arch project itself needs some process streamlining.
2. We need new features (higher level commands, mostly) to help
"share the love": to make it easier for new users to implement
good processes from the start.
Those are complementary. We can use the arch project itself as
a "sandbox" in which to develop new features that will answer
most of the common complaints from users. If we do that, we'll
also be more efficient and hopefully put to rest the contributor
complaint.
** Project Process Changes/Stable Branch
Each release attracts new users.
Frequent releases with lots of merged in contributions keeps
contributors happy.
Clearly, even though it is important that I be able to spend more
time working on new things (within arch and without), it is also
important that my going away to think and work on stuff not disrupt
the flow of merges and new releases.
At the same time, without wishing to denegrate any of the
contributors, none of the contributors are "good enough" yet
to have authority over what should be merged. Some of the
key contributors are very close, but I still find that I have
to edit contributions from just about everyone.
So:
(a) we need a separate "stable" mainline into which
small changes can be merged and from which frequent
releases can be cut
(b) we need to construct the process for the stable
mainline in such a way that I can review and approve
(or kick back for revision) contributions at a
very high rate, in spite of my slow modem.
In _theory_, this is a piece of cake with arch and setting
it up will make for a great model of "best practices".
In _practice_, what we have right now is most of the
infrastructure _parts_ for such a process, but scattered
around the room, in parts, awaiting consolidation and
assembly.
So there is some work to do to set up a stable mainline with the
desired properties and, if it's to happen anytime soon, I think it's
likely I'll need some volunteer help to create the infrastructure.
I therefore think it's reasonable for the GNU Arch project to
shift focus a bit and become a bit more "introspective" -- to focus
for now on its own internal needs and on solving those needs, using
arch.
As we do that work, at every step, we should be asking ourselves:
How can we implement the infrastructure we're building for the
arch project itself in such a way that users can easily emulate
that infrastructure in their own projects?
In other words, I'd like a closed solution and a generalized
solution: I'd like the entire infrastructure that we build to be
bundled with arch ("closed") and I'd like that infrastructure to be
general enough for people to instantiate for other projects
("generalized"). I stated above, under maintainer complaints, my
opinion that many user complaints are best answered by the addition
of higher level commands. Development of an improved infrastructure
for the arch project itself is precisely the testbed in which such
new commands should be thought out and then implemented.
Note that this approach makes building the new infrastructure harder
than it might otherwise be. If we didn't care about turning the
infrastructure project into new features, we could simply hack up ad
hoc solutions, make sure they run well on gnuarch.org, and leave it
at that. We could say to others "just duplicate the kind of work we
did". But since I do care about turning the infrastructure
project into the way new features are added, then we can't hack up
ad hoc solutions: we have to hack up clean, portable, and broadly
applicable solutions.
That ambition _may_ prove to be a problem. The risk is that the
infrastructure project goes too slow and so the solution to the
contributor complaint takes too long to arrive. I'll tentatively
assume that this problem won't occur: but let's also be wary that it
might and, if it does, we'll need to adjust the strategy a little
bit by first creating a _provisional_ infrastructure for the arch
project and then working on turning that provisional infrastructure
into new, polished features.
The bottom line: who (especially among established contributors) is
game for this? Can we muster more effort than I can provide alone
for persuing this strategy?
** Feature Plans and Plans of Attack
I shouldn't be the only one who "stops to think".
Remember, please, that one of my criticisms of the Subversion
project is that, when they should have, they failed to pause
and take stock before moving forward. The need to stop and
think for a bit is intrinsic to complex projects like Arch.
It's just plain healthy and should be a no brainer, from time
to time.
I have my own opinions about what the improved infrastructure for
the Arch project should look like. I'll briefly share some of
those here but:
I think that we should spend some period of time, let's say two
weeks, during which the new infrastructure is discussed and roughed
out. I invite everyone but, primarily the main contributors, to
help with this planning. What are _your_ ideas about what is
practical and desirable in the project infrastructre?
A few of my thoughts, particularly some that characterize the
"maintainer's perspective":
1. We need a standard for how to submit changes. Yes, yes,
if you tell me to cherry-pick this or that, or star-merge
from here or there, sure, I can do that. But having to
parse and implement every merge request by hand is a big
bottleneck. It would make much more sense to ask each contributor
to spend an extra 5 or 20 minutes preparing a submission in a
standard form than to ask me to spend 20 minutes or an hour
trying to figure out and implement whatever is meant by
an ad hoc merge request.
It's just a simple shift of labor. I don't want to cherry
pick contributions anymore or have to wade through weird merge
requests. Fortunately, arch means that we can take that
part of the work of merging and move it from maintainer to
contributor. It's easier for the contributor to do and, while
it means more work for contributors, overall, it's the more
efficient approach.
2. I shouldn't need to download proposed changes to stable. For
most _simple_ changes, I shouldn't even need to install them in a
project tree until I merge from the stable mainline into which
they've been merged.
Rather, I'll be very happy when I can visit a web page that
lists the submission queue, click through the diffs each
submission contains, click through to the results of an automated
merge of those diffs, click through to the results of a test
suite run of those diffs, and click to indicate my approval
or rejection, turning approvals into automatic commits to the
stable mainline.
I think that we can drop the time it takes to process a
simple submission from "most of an hour" to just a few minutes.
That's the difference between "something I'll postpone because
I don't want to spend most-of-an-hour on it" and "something I'll
do immediately because it only takes a few minutes."
3. It should be easier for casual contributors to contribute.
A platonic ideal might be for a user who has just a tree
that came from untarring a distribution to be able to say:
% tla fork
[hack hack hack]
% tla submit
and have their quick fix show up in the merge queue. The closer
we can get to that ideal the better, especially if it is easy for
other projects to provide a similar interface.
I think that the "FEATURE PLANS" I've posted so far, plus furth,
are rough specs for most of the raw functionality we need to satisfy
the three goals above.
The bottom line: Contributors, please stop yelling at me. Instead,
let's set up this infrastructure that will answer your complaints
and provide a good sandbox for answering the most common user
complaints. A good place to start is to help design the
infrastruture and then help to implement it. What are your
thoughts on those matters?
* Contingencies
In some sense, this roadmap reflects a fork in the road. It might
flop or it might be what we actually do.
It will flop if the only responses are more complaints.
It will be what we do if some of the current complainers step up to
work on the barn raising.
* Release Map
Release Arch 1.3 should be made soon, prior to any infrastruture
changes. It should include the security patch from 1.2.1 plus
all trivial fixes and improvements in the current merge queue.
I wonder if one of the established contributors would be interested
in preparing release 1.3, but subject to a particular requirement.
In particular, would any of you (who know who you are) be willing
to prepare a 1.3 branch with the changes for the release divided
into clean, separate changesets? In other words, to make it easy
for me to just read the changesets, one by one, veting each before
moving on to the next? This would be close to a "by hand"
simulation of the hoped-for stable branch.
Thereafter, we can maintain:
1.3.X --- the stable branch
1.4 -- the development branch for the advanced features
needed to create our infrastructure
* A Reminder
Just because I spend some time working on things that don't
immediately produce a new release or carry out a new merge, that
doesn't mean I'm not busy doing things that contribute to arch and
my other projects. Really, it's almost the opposite: mindlessly
merging and cranking releases would not be much of a help to the
arch project; working out good plans and then getting them started
is a big help, I hope, for the project. I try to keep my work at
least minimally visible via the public mirrors of my archive.
GNU Arch is a community supported project, at the moment, both in
terms of labor and in terms of money. The project exists today
because of contributions from users like you and I (and satisfied
arch users everywhere) are grateful for your support. Your support
allows me and the community generally to keep arch going and
improving as we continue down our path of simply and unambiguously
_taking_over_ the space of revision control tools on the strength of
our fundamentally superior technology. It is also no small pleasure
to work on a project that promises to correct the intrusion on the
free software world by BitMover.
One of the best ways to help the Arch project continue to progress
(and to help my other projects) is with financial contributions:
https://www.paypal.com/xclick/business=lord%40emf.net&item_name=support+for+arch+and+other+free+software+efforts+by+tom+lord&no_note=1&tax=0¤cy_code=USD
and
address@hidden for www.moneybookers.com payments.
-t
- [Gnu-arch-users] Arch Roadmap Draft (the anticipated part 3),
Tom Lord <=