Re: Presentation: "Publisher-grade LilyPond" in Ottawa

[Carl Sorensen]

On 6/13/10 12:02 AM, "Boris Shingarov" <address@hidden> wrote:

> Hi Graham and all Lilyponders,
> 
> I was quite astonished to hear that my slides were understood to mean anything
> pessimistic or negative.  If they give people this impression, then it is a
> defect of the slides which I will fix.
> 
> But right now, let me address some of the apparent misunderstandings.
> 
>>  
>> Are you seriously going to submit a paper to CMJ saying "a volunteer
>> open-source project has limited resources for fixing bugs" ???
>>   
> This is not what we are saying in our presentation and/or paper.
> What we are saying is: "We attempted a publication of a major critical edition
> through a major publishing house, using software from a volunteer open-source
> project with limited resources.  We first hoped that this software would be
> immediately (or almost immediately) suitable for critical-edition work.  We
> found issues blocking this work.  These issues are orthogonal to the main
> direction of the LilyPond project.  We fixed them, making the book possible. 
> Future work includes making these solutions useful for the wide LilyPond
> audience, not just for the immediate needs of this particular book".
> 

I like this message.  However, as others have noted, I didn't get that
message from the slides.


>>  
>> Oh, and I hope that this "paper to appear" includes an excellent
>> reason why you didn't use lilypond-book and LaTeX, which unlike
>> lilypond itself is *designed* to mix music and text.
>>   
> Two reasons, really.
> 
> Reason one, we did investigate that route.  Lilypond-book and LaTeX do not
> achieve the kind of music/text integration that the editor demanded.  Another
> issue was integrating with their existing database of variant readings.  And
> for me, there was the obstacle that it was not what the end user was asked
> for.
> 
> Reason two, the cry for help was heard all over the mailing list starting
> many, many months ago.  Anyone with a LaTeX-based solution yet?  Anyone even
> *suggesting* a LaTeX-based solution yet?  After a year and non-trivial
> (thousands of Euros) bounties offered?

Where are the thousands of Euros bounties for the LaTeX-based solution?  I'm
not aware of *any* thousands of Euros bounties for anything on LilyPond.
This is a serious question, not a rhetorical question, by the way.  I have
looked at the issues with Bounty tags, and the only one I can find that
seems to be relevant to this discussion is "support for footnotes and/or
endnotes".  That issue makes no mention of a LaTeX-based solution.
 

<snip>

> 
> "The community showed no interest in addressing these issues", well I am
> certainly not saying it's the community's *fault* -- the point is that there
> are other dimensions, which are of concern to [at least one] top publisher, in
> connection with [at least one] type of book, and which are orthogonal to the
> dimension of the direction of work of the LilyPond project. 

I think that "these issues are orthogonal to the current work in the
LilyPond project" carries a much different connotation than "the community
has no interest in these issues", even though they may cover the same facts.

> After all, none 
> of these issues originated from me; they originated from the musicologist
> doing the chant research, and the editor of the book -- before I even started
> looking at LilyPond -- and were rejected as not even being valid LilyPond
> problems.  The typical answers were,
> 
> "I think that you'll be better served by others"
> http://www.mail-archive.com/address@hidden/msg52031.html
> 
> "Your post is absolutely unnecessary"
> http://www.mail-archive.com/address@hidden/msg52334.html
> 
> "I recommend magic"
> http://www.mail-archive.com/address@hidden/msg52026.html
> 
> Does this represent a problem enough to "discourage" people?  I think it does
> not, for two reasons.  First, people are smart to take such things with a
> grain of salt.  If you look through the lkml archives through the last decade,
> you will see a lot darker atmosphere in that community, yet there are plenty
> of active kernel developers.  Second, and I insist to repeat it over and over,
> the main LilyPond project focus, and what we are doing in our
> "Publisher-Grade" project, are different, orthogonal dimensions.

May I suggest that "Publisher-Grade" is perhaps a misnomer?  This makes it
sound like we can't produce *any* publisher-grade music without footnotes,
or endnotes, or widow/orphan control.  Perhaps an adjustment to the title to
"Publisher-Grade Critical Edition" would help clarify things.

> 
> "Patches were hard or impossible to contribute back" -- this is through no
> fault of the community.  In fact, working with Joe Neeman -- both to initially
> produce patches, and to make them conform to the LilyPond standards so they
> can be pushed into the main codebase, -- has been a highest pleasure, and his
> help and guidance are priceless.  His time and patience are highly
> appreciated.  The help from other developers, has also been great (Neil, Carl,
> my sincere thanks!!)  But this is not what I am talking about; all this work
> together with Joe and others, it's all about "the LilyPond dimension", while
> the problems blocking the publisher, are in those other dimensions, and
> LilyPond being a volunteer project, no one has incentives to do that
> not-very-core, not-very-exciting work.
> 
> So it's a question of the proverbial "scratching one's own itch".
> 
> Simplest example: a patch fixes a bug (a Blocker for our real-life project). 
> The fix is used in production for some time, and seems to be working fine. 
> Code review on Rietveld, patch looks good to the reviewers.  The only problem
> delaying its push, is the absence of a test case.  Ok, I spend some time
> adding a test case, but bump into problems.  Joe is so kind to offer his help
> in debugging; but to get through the debugging of the problem would seem to
> require half a day, maybe a day.  Do I understand the importance of having a
> test case?  Yes!  Do I want a test case?  Yes!!!  But it's about customer
> value. "A defect is anything that leads to customer dissatisfaction".  Well,
> the end-user does not care about having a test case.  The time is very tight,
> and we have other pressing priorities (like other Blocker bugs).  He would
> appreciate the inclusion of the work in the main LilyPond codebase, but not to
> the extent of stopping work on the other Blocker bug in order to write the
> test case so that the fix gets pushed to the main repo.  If it's an hour,
> yes.  But five hours...let it sit in the vendor tree.
> 
> There are also patches that require a lot more work to become universally
> useful to all LilyPond users, not just to our book here.  When a patch takes 3
> days from start to deployment in production, and half a day to polish to be
> pushed into main repo, that's fine, I do that half-day.  When a patch takes
> the same 3 days, and would need another 5 days to be able to conform to
> LilyPond's standards, I might, but the customer would not appreciate it, so
> *having* the patch would in effect constitute a defect.
> 
> What's the way out of this?
> Well I *hope* that by solving these "other-dimension" problems, we will be
> able to make LilyPond a platform for serious projects which will support its
> development.  That way, we will be able to afford real professional
> development; afford full-blown solutions, to meet all standards, and be useful
> to the community at large; get everything from my vendor tree into the main
> tree, and have no need for a vendor tree any more.  This is what I meant when
> I wrote that last "Future Work" slide.

I'd appreciate anything you could do to clarify (for the  -devel list and
for the bug tracker, not necessarily for the paper) the status of the
"vendor tree" relative to inclusion in LilyPond.  For example, if there's a
patch available that is just missing its test case, then we ought to have an
issue about that.  If there are other patches that are not yet up to
LilyPond status, but might be able to get there, it would be nice to have
them listed as issues with patches attached.  Then we'd at least know that
there is work to be done.  As long as the patches are only in your vendor
tree, the community can't really help.

> 
> On 06/12/2010 11:28 AM, Reinhold Kainhofer wrote:
<snip> 
>>  
>> I first tried lilypond, too,
>> but soon switched to latex. I don't have to mix music with text, but only
>> insert some figures
> That's the thing -- we HAVE to mix music with text, and because if the
> specifics of the plainchant we are dealing with, fragments HAVE to flow
> seamlessly with the text.
> 
>>  
>> I don't think there is zero interest in having the problems fixed, rather the
>> contrary. However, there is zero time to address them ourselves.
> What I am trying to do, is create some sort of professional LilyPond
> ecosystem, where people would be allowed to spend serious amount of time on
> LilyPond work, but where problems would actually get fixed.  If a publishing
> project is willing to spend many thousand dollars to fix a certain problem,
> and the only kind of response they get in a year, is "I recommend magic" and
> "Your post is absolutely unnecessary", that kind of
> interest-in-having-the-problems-fixed does not really matter to the publishing
> project.
> 

There are two ways to ask for help.  One way is to ask for somebody to
implement a feature.  The other way is to ask for guidance in how to
implement the feature oneself.  I'm not aware of any "guidance" requests
that have gone unanswered.  On the other hand, there are *lots* of feature
requests that go unanswered.

>>  
>> And, btw, I'm very interested in footnotes, as I'd need them with my critical
>> editions, too...
> At this point, we temporarily put them on the back burner.
> When the question about footnotes was asked on the list, no one raised their
> hand saying "I'll do it".  Plus, footnotes would be probably less good in the
> book we are preparing than endnotes, because of the huge amount of variants. 
> If we place the critical apparatus in footnotes, it will occupy almost the
> whole page, making the book difficult to use for practical singing.
>

Nobody *ever* raises their hand and says "I'll do it" for a feature request
in a new area.  If I had just asked for somebody to implement fret diagrams,
I doubt they'd be part of LilyPond yet.
 
>>  
>> BTW, have you fixes for the vertical alignment been pushed yet?
> You mean, the vertical space estimation?  There were several bugs screwing
> height-estimation.  Some of those fixes are pushed, and some are still sitting
> on Rietveld.  Which ones specifically were you interested in?  Or do you mean
> the text / embedded score alignment?  The fix to that, requires as a
> prerequisite the "multi-line embedded score" fix which is sitting on Rietveld.

Can you give us Rietveld issue numbers?

Thanks,

Carl