GOP2-3: GLISS (update 1)

[Graham Percival]

There seems to be fairly broad support for _some_ form of
standardization.  Here's an update of the proposal along those
lines, along with brief responses to common concerns, in order to
let people just joining the discussion to skip the past 50 emails.

Better formatting here:
http://lilypond.org/~graham/gop/gop_4.html

** Summary

Let’s start stabilizing portions of the LilyPond input syntax. We
will guarantee that selected elements of the syntax will not
change (even with convert-ly) during the 3.x releases. This will
be a slow process, and the first phase (2012) will not even cover
the entire “single staff notation” section in the tutorial.


** Motivation

Some “computer languages” are fairly stable. A TeX or C++ program
written 10 years ago will probably still compile with no
modifications (notwithstanding the g++ 4.3 header and namespace
changes). The same is not true of LilyPond; even after using
convert-ly, there are still bits that require manual updating.

Given that, LilyPond is not suitable as an archival format for
music. It can produce a great pdf when you first write the file,
but the .ly files require regular maintenance if you want them to
compile in the latest stable version of lilypond. This is a
problem for projects such as mutopia – a large fraction of their
.ly files don’t compile with current lilypond. That means that
they can’t benefit from recent bugfixes; users wanting the sheet
music in a different size (say, printing a choral score as an A5
booklet) must delve into the ly code and make manual changes.

A stable input syntax should also make it easier to write
converters to/from lilypond, and should also make it easier to
write GUIs for lilypond. Currently, any program which exports
lilypond code needs to support multiple versions (e.g., 2.12 vs.
2.16). Hopefully making it easier to output lilypond code will
lead to more/better programs which do this, either in terms of
converting from alternate formats into lilypond, or in terms of
GUIs calling lilypond as the backend.

On a personal note, this is one of the biggest reasons I’ve given
up on using lilypond personally. From 2001 to 2004 I got a minor
in music composition. I carefully kept all my .ly files but
foolishly did not preserve the pdfs. And now, 10 years later, I’m
left with a bunch of music that I cannot generate sheet music for.
It’s true that I could dig out old lilypond binaries to process
the ly files (and I’ll probably tdo that at some point), but it
would be much nicer if I could benefit from the past ten years of
bugfixes in lilypond. Manually updating the .ly files would take
hours or days; I’ve started this process a few times but always
lost interest after a few files, since there’s no guarantee that I
wouldn’t need to go through the same process in another few years.


** Why disallow convert-ly?

  -  it forces us to take the process seriously by removing the
"safety net". Any poor decisions from the process will be
enthroned in the syntax for years to come[1]. Hopefully this will
make us proceed cautiously, take a more serious look at the syntax
proposals for potential problems, etc.
  -  it signals to other projects that we’re serious about this.
This makes tasks such as writing importers/exporters to/from
lilypond much less undesirable. It also might help people doing
musicology (or music theory) research with lilypond files.
  -  it makes lilypond more suited to being an "archival" format
(or at least less unsuited). convert-ly only converts files in a
forward direction. Granted, there aren’t many instances where
somebody might have a corpus of music they want to render in both
lilypond 3.0 and 3.2, but it’s not impossible. For example,
suppose there was a team of a dozen Russian musicologists
archiving folk tunes, but lilypond 3.2 doesn’t work on OSX 11.4
because Apple broke their own API again. It would be nice if the
team could share lilypond files between lilypond 3.0 and 3.2.
(assuming that there were no special tweaks happening – i.e. the
team was first getting the notes and rhythms written down, and are
not planning to do a great deal of tweaking). 


** Will this help the parser?

Straightening out the parser is going to lead to some breakage in
complicated and/or badly written scores. That may lead to some
understandable frustration from some users, but if we’re running
GLISS at the same time, that gives them some hope that things will
settle down. Also, simply discussing the notation we wish to
support will give rise to questions about precisly what the
current system already supports, and can clarify our thoughts on
it.


** Not necessarily any changes

GLISS will not necessarily involve any change of notation; in
fact, the first portion of “syntax stabilization” could just end
up approving the existing syntax exactly as it stands. I think we
should discuss each notation element separately without simply
rubber-stamping the existing syntax. If there are any changes in
the basic notation, then of course it would be extremely bad if
convert-ly couldn’t handle it. But the end result of such
discussions could lead to the conclusion that any disadvantages of
the current notation outweigh the pain of changing. The important
thing is informing users (and programmers) about what elements are
guaranteed to work for every 3.x release.


** Subset, not complete definitions

When we discuss something like accidentals, we’re not committing
that the finalized syntax will be the only way of achieving the
relevant goal. For example, if we standardize on cis to indicate a
C sharp, then at a later date we can still introduce syntax such
as c+1/2 to indicate the same C sharp (the latter possibilities
are to allow more greater flexibility for microtonal notation).


** Multiple rounds of standardization

It’s really easy to under-estimate the work that goes into such
discussions and implementing the changes. I think we should start
very small and expand gradually. At the present time, we will
decide on what to do in the first round only, but I’ll add a very
tentative suggestions for rounds 2. There will be another GOP
discussion to settle on the exact range of notation tackled in
round 2.

The basic idea is to spend approximately 3 months discussing some
changes, then spend a few months implementing the changes (with no
syntax discussions). Then we’ll have a stable release, and wait
for at least 6 months to see of there’s any complaints with the
input syntax; if there’s no complaints, then we’ll declare that
part of the input syntax to be “finalized”.

  -  Round 1: note pitches (absolute and relative, common 12-tone
accidentals only), durations (powers-of-two and non-nested
tuplets), key signatures, time signatures (non-compound), bar
checks, dynamics, partial measures and grace notes. Also setting a
title and composer, and outputting in pdf and midi. Expected to be
implemented by the end of 2012 and finalized by the end of 2013.
  -  Round 2: ties, slurs, articulations, adding text, manual
beaming, chords, multiple voices on a staff, multiple staves,
lyrics, variables/identifiers. 


** Subset for first phase

In greater detail: I’m suggesting that we have multiple rounds of
syntax stabilization. The proposed elements of current lilypond
notation which we will stabilize is captured by these two files:

\version "2.16.0"
\score {
  \new Staff {
    \new Voice {
      \partial 8 d8 |
      c4 d' e, f'' |
      \times 2/3 {a4 b c} \grace {d16} d2 |
      \acciaccatura {b16} c2 \appoggiatura {b16} c2 |
    }
  }
  \header {
    title = "don't overwrite this title"
    composer = "the lilypond GLISS team"
  }
  \midi {}
}

\version "2.16.0"
\score {
  \new Staff {
    \new Voice {
      \relative c, {
        \clef "bass"
        \time 3/4
        \tempo "Andante" 4 = 120
        c2\mp e8 c' |
        g'2. |
        \time 6/8
        \key d \major
        \tempo "Allegro" 4. = 120
        f4.\f eisis8 eis r |
        deses,8 des r e'8 c c,8 |
      }
    }
  }
}

and then we guarantee that these files will compile, completely
unmodified (no convert-ly allowed), for every lilypond 3.x
version. This seem like a really small step, but I really don’t
think that we can overestimate how much time, energy, and
arguments this will require.


** Example questions

Here’s a few sample questions that we’d encounter even with a
really small subset.

PLEASE DO NOT DISCUSS THESE RIGHT NOW.

  -  do we keep dutch as the default language, or switch to
english?
  -  do we finally make that \times -> \tuplet change that’s been
discussed for years?
  -  \score \staff vs. \new score \new staff.
  -  what’s the canonical input structure? what shorthands do we
commit to supporting? 


- Graham