Re: [Bug-gnupedia] Design proposal

[Bryce Harrington]

On Tue, 23 Jan 2001, Bob Dodd wrote:
> Hi Bryce,  um... where to start?

That is sort of the point.  The starting point is rather undefined (or
at least, only roughly defined.)  We must have something to start from,
that we can use NOW.  Else it is going to take forever to get anywhere.

How do I know this?  Because I've done it.  Let me explain.  I am the
project coordinator for another project, called WorldForge, which got
its start from a Slashdot announcement.  A rough vision was held by the
many people who joined, but there was a great deal of disagreement over
the requirements, plan of attack, and how to coordinate various
efforts.  There was a good deal of petty bickering that almost got to
the name-calling stage at times.  

We started from ground zero.  We didn't have *any* code, didn't have a
design document, and not even a set of requirements laid out.  When
people did try to put out such things, they were ignored and the author
abandoned them in frustration, uncertain why they didn't receive
critiquing, or if critiqued, why the criticism was so dismissive.

Fortunately, we had a number of people dedicated to the overall goals,
and skillful in a wide variety of areas.  Progress was being made in
isolated, independent corners of the project.  But the problems
persisted.  Finally we decided to forego trying to do everything from
the start, and instead start with something very simple, and bootstrap
our way up, undertaking successively more challenging game objectives.
The first game wasn't even a game, just a server you could connect to
and move around and chat with others.  The next game was had a little
more gameplay but still was far, far, far from what our real objective.
There were concerns that by taking so many shortcuts that we were
channeling ourselves into a blind alley, that we would be make bad
choices early and be unable to escape them.  Or that by having something
that "almost" worked, it'd sap energy and efforts (i.e., "investing time
into something that will be thrown out in the end") that could have gone
to the 'real thing'.  

But on the contrary!  Having something that worked (except for foo and
bar...) was *extremely* stimulating, because it was immediately clear
that foo was missing and desperately needed, and that bar actually was
not that big of a deal.  And being able to focus on _finishing_
something, and get it delivered, was very stimulating to ourselves, as
well as to others, who joined to see what was up.  The bickering died
down (well, shifted away), people started seeking out ways to cooperate,
rather than simply argue, and there were always answers to the question,
"Okay, what can *I* do?"

It is always exciting to start something from scratch.  Then you don't
need to bother learning how others have done it.  Novice programmers
especially like the idea of skipping the boring research and diving into
the sexy guts of the thing.  The Journeyman knows that this falls apart
once you get a few thousand lines into the thing, and declares that
there must be a thorough set of documentation before any coding should
be allowed to proceed.  The Master realizes that while sometimes there
is a time and place to reinvent the wheel, usually you can make do with
something that exists - especially if the thing is not central to the
project.  As one master once told me, "That sounds like a wonderful plan
for a sawhorse, but are you really in the sawhorse-making business?"

The above is my experience, and why I personally think *this* project
needs to adopt some existing piece of software (like Wiki, as I
suggested, but I know there are probably many others to choose from), or
an existing encyclopedia (like the one in the Gutenberg Project), or
hell, just put up a raw web directory to show articles that have been
submitted.  I posted a 'battleplan' a few days ago to illustrate exactly
how such a bootstrapping program could be done here.  It's been a little
funny that each has either been dismissed or patronized.  But also kind
of discouraging.


Well, now maybe you still think I'm full of it.  Okay, then don't take
my word for it...  ESR has said exactly the same thing, in his paper
"The Cathedral and the Bazaar":

http://www.tuxedo.org/~esr/writings/cathedral-bazaar/cathedral-bazaar/x285.html

To pull out just one quote:
"It's fairly clear that one cannot code from the ground up in bazaar
style.  One can test, debug and improve in bazaar style, but it
would be very hard to originate a project in bazaar mode.  Linus
didn't try it.  I didn't either.  Your nascent developer community needs
to have something runnable and testable to play with."

If you have an allergy to ESR, then look at RMS's announcement of this
project, where he makes very similar points.  "When a project is
exciting, it is easy to imagine a big contribution that you would like to
make, bite off more than you can chew, and ultimately give up with
nothing to show for it."

He clarified and reiterated it with a post Monday on this.
"Once encyclopedia work reaches the point where there is enough
experience, then it will be useful for people to look at how work is
done, and figure out what sort of tools could make it easier."  He
cautions against writing special tools before we know what the heck we
need.  And I totally agree.

I've been through one project that has attempted to originate in bazaar
mode.  It can be done, but it's a bitch.  And IMHO there's no reason we
have to do that.

 
> Well, I guess I would start by saying that we need a very clear
> definition of the requirements of this here encyclopedia thingy. 

I agree that requirements are always a wonderful thing to have.  In my
opinion, they're even more valuable than the design, because once you
have achieved group concensus on what the goal is, the design tends to
fall out pretty easily.

That said, I have seen wonderfully detailed requirement documents fall
on their ears when everyone runs away from what appears to be "work".
Or people will mix up requirements and design.

I think that there may still be some disagreement over the basic
_concept_, still.  It's hard to gain concensus on a set of requirements
when the concept is still controversial.  

> Coming back to the Wiki, I think Wiki's are a great idea for
> collaborative writing projects when the basic content structure of the
> material is well udnerstood. 

I have worked with wiki for about 3 years now, on 6-8 different
projects.  I was one of the reviewers of the Wiki book about to be
published by Addison Wesley.  Wiki's work well even when the basic
content structure is not well understood, because they allow structure
to be organically shifted around as needed.  

> For example, it would be a fantastic
> medium in which to write our requirements document, if Hector would be
> so kind as to install one on our website. 

I believe you are welcome to post it on www.wikipedia.com and use their
wiki.  They have even set aside a section of it for this project (under
GnuStuff on the Homepage).  Setting up your own wiki is not really that
hard, though.  I guess my thinking here is "A bird in the hand..."

> As an encyclopedia tool, I have more doubts, regardless of our detailed
> requirements. Do you think the Wiki is providing just the basic "core"
> organisation of the encyclopedia, or do you think it's also providing
> all the different views and search engines too? 

It can provide different views (if I understand 'view' to be the same as
"index").  There are many ways to incorporate search engines to wiki's,
however I believe until there are thousands of articles any old search
engine will probably do fine.

> So, are we talking about the same Wiki (and its front end) for
> inputting information, and for searching?

Wikis can be integrated/extended with other scripts quite easily.  The
input mechanism can also be reconfigured as desired.  

> How would we deal with people wanting to access content without using
> the existing Wiki front end?

Wiki's store their data either in a database or as flatfiles (I've seen
it both ways).  There is nothing complex about how it is stored.  For
mere viewing, there is nothing to stop adding other front ends.  For
viewing and editing, you would need to include appropriate mechanisms to
lock between the wiki view and your new one (which of course you plan to
put into this new mechanism, anyway, right?  *grin*)

> How would people be able to
> filter content (e.g. to make a "children friendly" view that excludes
> material inappropriate to the age group) 

I have seen several different ways to do filtering.  a. Some wiki's
allow segregating data into different categories, each with their own
htaccess files, thereby allowing closing out access to certain areas of
the site.  For instance, we use this mechanism at WorldForge to prevent
general access to the adminstrative portion.  b. Some wiki's allow
identification of "secrets" within a page.  Only people logged in with
the correct permissions will see these secrets; others see no trace of
the missing material.  c. Most wiki's are pretty easy to extend with
unique tags, filter criteria, etc.  So during the render process, you
can include a check for swear words, or scan for a tag in the source
indicating there is "NAZI-Propaganda" contained in the page, or
whatever.  

> How scaleable is the back-end?

Ah, you missed my pointing out of "InterWiki" yesterday.  ;-)

Wiki is *extremely* scalable.  To start with, by itself it can be
configured to be very low-impact.  For example, frequently accessed
pages can be cached and only periodically generated, so that these
accesses maximize your web server's static-serving capabilities.  In
fact, every page in the site can be staticized.  You can even have
"inner" and "outer" versions of the site, where the inner is dynamically
generated and editable, and the outer is a periodically updated static
copy.  

But this is *way* overkill to anything this project would need for a
long time.  Note that most wiki scripts are very short.  KeheiWiki
(used for a number of large-scale commercial sites) is fully featured
yet weighs in at only a hair over 1000 lines.

> How does it compare to other free encylopedia-line information systems
> (e.g. Nupedia's own source code) 

Zope - this is a Python based system for managing sites.  It's claim to
fame is that it is object based and extremely configurable.  It is not
geared towards encyclopedia-type usage but could be customized to do
this.  A nice feature of Zope is that you develop right *in* zope, and
can test new functionalities out directly.  It has support for a WIDE
range of document types (including wiki!)  I've not suggested it until
now because I believe it to be a shade beyond our present capabilities
to implement.

Everything2 - This is produced by Slashdot and its engine was frequently
suggested by the Slashdot folks.  I think this was inspired by wiki, or
at least it emulates much of its functionality.  I do not know if it has
near as wide of usage as wiki, so it might be more in the "toy" class,
however it would probably be worth comparing to.  

Various PHP based solutions - If you take a gander at Freshmeat and a
lot of the "knowledge-oriented" sites around the web, you'll find PHP
behind most of them.  I don't have any firm opinions about these, except
that they tend to be written fairly specifically to the task at hand
(which is commendable in and of itself, but suggests if we went this
route we'd need to emulate rather than reuse.)

> Unless we can answer all of those questions (and many more), I don't
> see how we can choose *any* existing tools or platforms.

I hope I have sufficiently answered all of the questions (and likewise
hope I haven't bored everyone else to death.)  It is not my intent to
rush any judgements here, although I'll point out that wikipedia already
exists and is gathering articles into it daily.  Understanding the
objectives, and then researching the alternatives is a very good
approach to take.  To me, it is the encyclopedia articles themselves,
moreso than the software they reside in, that I care about.  

And remember, I am suggesting wiki *only* as an intermediate to get us
started.  I would not be so bold as to try to promote it as the *only*
solution, forevermore.  It is arrogant to think that we're smart enough
right now to know the perfect solution.

As I said above, I think this was the point RMS made in his email
Monday.

Bryce