Re: Issue 2702 in lilypond: Patch: Unify the lexer's idea of words and commands across all modes.

[David Kastrup]

Graham Percival <address@hidden> writes:

> On Sun, Jul 29, 2012 at 10:15:10PM +0200, David Kastrup wrote:
>
>>  It turns out that this definition works with  
>> both "make test" as well as "make doc" without requiring any change in the  
>> LilyPond code base.
>
> That's certainly promising.
>
>> Discuss.  This is quite a consequential change regarding what word syntax  
>> is valid and what not, but the previous state was rather arbitrary to the  
>> degree of being wonky.
>
> Could I have some examples?  I just don't get this "word"
> business.  Is there any syntax which was previously
> (theoretically) supported, which this patch breaks?

Here is one thing that can be made to work: we can have music inside of
output definitions, but currently it is parsed in "initial" mode.  That
means something like

\layout { \tempo 4. = 120 }

will not work and has to be written as

\layout { \tempo 4 . = 120 }

since 4. is a real number in "initial" mode.  One way out would be to
switch into "notes" mode temporarily for music.  If we do that,
something like

\layout { \tempo "Moderato"
          line-width = 100\mm }

will not work any more, since

          \tempo "Moderato" 4. = 56

is perfectly valid input, so the parser has to look ahead to see whether
anything that might be a duration will follow.  It can't switch back to
"INITIAL" mode for that.  But with our current setup, a "word" in
"notes" mode will not be allowed to contain hyphens, so the next token
is a STRING with the value "line".  In "INITIAL" mode, it would be a
STRING with the value "line-width".  Having just "line" be the string
means that the assignment goes haywire.

In contrast, if words have the same form in all modes, "line-width" will
get scanned correctly even while still being in "notes" mode, and the
following assignment will work properly after switching back to
"INITIAL" mode nominally one token late.

Having fewer different modes in the lexer also means that mostly
mode-unaware tools like highlighting editors and convert-ly are more
likely to get the basic units they are messing with identified
correctly.

-- 
David Kastrup