Obstacles to LilyPond Development (was Re: state of the release: the Good, the Bad, and the Ugly)

[Carl Sorensen]

On 6/1/10 1:26 AM, "David Kastrup" <address@hidden> wrote:

> Carl Sorensen <address@hidden> writes:
> 
>> On 5/27/10 7:05 AM, "David Kastrup" <address@hidden> wrote:
>> 
>>> It does not help with explaining the internals of Lilypond all too
>>> much, but then nothing does right now.
>> 
>> In my opinion, the lack of a clear explanation of the internals of
>> LilyPond is the biggest obstacle for getting started in development.
> 
> I am not all too sure.  For starting development, it is more important
> to know what you _don't_ need to touch for a given task than what you
> do.  And if you do need to touch a lot, then it may be a sign that the
> code (and its responsibilities) is not organized well.
> 

I don't think we disagree here.  Since there is no clear explanation of the
internals, a beginner has an almost impossible task to figure out where to
work and where not to work.

>>> In a healthy project, contributors come and go.  If you instead only
>>> see the same faces leaving and returning, something is amiss that
>>> makes starting to contribute hard.
>> 
>> Yes, this is true.  In your opinion, what are the top 3-5 things that
>> make starting to contribute hard?
> 
> Properties.  There are tweaks, overrides, sets.  Some of them work on
> some properties, and there is no user level coherence to what you need
> to do on what and why.  Yes, I had some fits about that already, and
> some people repeatedly told me I am an awful child for keeping up the
> "why, why" questions and that things were just so.  But I am arrogant
> enough to say that something that can't be explained to me in a way that
> I understand it is a mistake in a programmer interface.  And we are
> talking about a _user_ interface, one you can't avoid using.
> 

I know that properties is a major issue for you.  I think that it's
primarily a user issue, rather than a developer issue.  I'll try to give a
brief answer in a separate email, because I don't want this one to get
hijacked.

> Write-only code.  If you try figuring out what the engravers and
> performers an contexts do and look in the code, you'll see a maze of
> twisty little passages and convenience macros for declaring things.
> What you don't see are comments.  If "outsiders" contribute anything,
> they are prodded to simplify and document their code until a "regular"
> can look over it and understand it without trying.  There is some nice
> idea behind it, but it turns into a failure because of several reasons:
> a) the resources available for review from the "regulars" are scarce, so
> only trivial code can pass their examination.  That means that
> non-trivial extensions of Lilypond remain the privilege of regulars.
> b) the regulars don't apply the same standard to their own code.  Or
> rather, they consider their code fine when _they_ feel they understand
> it.  After all, their time is scarce.
> The resulting code is maintainable only by regulars: either because it
> is trivial, or because they wrote it themselves without investing the
> effort to make it maintainable rather than just working.
> 

I agree that the code is *extremely* difficult to understand.  I'm not sure
yet if I know whether it's a "maze of twisty little passages, all alike" or
a "twisted maze of little passages, all different".  But that maze reference
brought back my feelings when I first started looking at an engraver.

Do you have any suggestions about how to fix this issue?  In my mind, I'm
hoping that the Contributor's Guide, sections 9.1 and 9.9-9.11 will
eventually serve as a map through the maze.  I realize it would be better to
rebuild the maze, but I don't think we have the resources to do so.

> Complete lack of documentation of Lilypond's structures.  Instead, one
> is pointed to a diploma thesis centered about music streams, a concept
> that does not actually exist in Lilypond.  The thesis talks about
> engravers returning a flag to indicate whether they accepted a certain
> event.  The respective functions in the C++ classes are void: they don't
> have the option of accepting an event.
> 

At one point I tried writing documentation of LilyPond's structures.  My
effort was unsuccessful.  You can even see the citation in Erik's thesis
(with a mention that it's useless).  As I mentioned above, I hope that the
material in Contributor's 9 will eventually grow into effective
documentation of LilyPond's structures.

> Complete lack of consideration for Midi: there is no way in the world
> that a user, or programmer can design and implement some engraving
> facility and also implement some Midi results: all manuals are
> effectively silent about Midi: there is no information at all about how
> to do _anything_ in that area.  But that means that most contributions
> are deficient: they only lead to visual results, ignoring the audible
> ones.

Most of this was a design-time decision.  LilyPond was intended to be
primarily an engraver.  Midi output was intended to be an afterthought; its
only intended purpose was for proofhearing, i.e. to see that the note
pitches were correct, and the note lengths approximately correct.

I see this as a deficiency in LilyPond, but not as something that makes it
hard to contribute, unless your contribution wants to be to improve the Midi
capability.

I doubt that anybody would be turned away who wanted to improve LilyPond's
Midi functionality.  But I also expect that there are less than 6 people in
the world who really understand the existing functionality.

> 
> Complete lack of consideration for Scheme in Lilypond's processing flow.
> Scheme is used in a callback manner, but everything worth happening is
> tied together using C++ classes.  Opaque C++ classes, without Scheme
> equivalents.  Considering that Scheme has been made part of Lilypond in
> the first 10% of its life span, that's not really impressive.
> 

I'm not sure I understand this comment in its entirety.  If it were up to
you, how would you change this situation?

Thanks,

Carl