Re: preliminary GLISS discussions

[Han-Wen Nienhuys]

On Sun, Sep 2, 2012 at 8:24 AM, Jan Nieuwenhuizen <address@hidden> wrote:
> David Kastrup writes:
>
>>> Maybe the time has finally come to drop convert-ly and implement and
>>> fully supported conversions using LilyPond on music stream level.
>>
>> You still need a parser of the appropriate version at the front end.
>
> We have perfectly fine ly parsers of each available version available in
> executable form from lilypond.org.  What we do not yet have, is a handy
> or integrated way of dumping the music tree with the original binary
> [read: a nice web service -- this could possibly be integrated with a
> mutopia revival, I'll be looking into this] reading the music tree with
> the current version and a perfect ly-dump function.  Eg, I think we may
> want to preserve %-comments in the music tree, or other stuff the user
> does not want to lose?

The tree is not what people want, though?  The tree has no information
about identifier subsitutions, and you only get the output of each
music function application.

You could argue that you could use the LilyPond parser to understand
the file for the rewrites. Unfortunately, due to how LilyPond works
(eg. the funky way music functions are parsed), you have to run the
full parse to know how to interpret something:

  #(set! identifier (..complicated expression .. ))
  \relative \identifier { a b }

you have to know the value (ie. the type) of \identifier, to know if {
a b } is the music relativized (yay for optional arguments). That in
turn implies running all the inline scheme. At this point, you are no
longer just "parsing" the input, you are actually processing it.

Since the output of the parser is not good enough (remember, we're
applying destructive operations during the parse), you have to modify
the parser to store all kinds of intermediate state about how tokens
are interpreted.

I am not saying it is impossible, but it is a lot of work, and you
don't get this type of thing "for free" by any measure.

I think it shows one of the fundamental weaknesses of how we
implemented the parser. The traditional model is the following:

1. Parse entire program into abstract syntax tree
2. Process tree to execute/compile the program.

Syntax transformations are easy: you can have a simple parser to read
the program into a tree, do some rearrangements on the tree, and the
serialize the tree back to a program. Successful examples of this are
Python (the AST package) and Go (packages go/parser and go/ast)

In LilyPond both phases are intertwined, since at any point during the
parse, we escape to Scheme to do execution, and the results of the
execution affect how the rest of the .ly file is parsed. This makes it
very difficult to do any kind of syntactic transformations
programmatically.

To me, a Grand Input Syntax "fixing" of LilyPond, would amount to
creating a syntax that strictly separates parsing and interpretation.
This implies not only rethinking  a lot of syntax, but also it means
letting go of some of the flexibility and conciseness of the current
format.

--
Han-Wen Nienhuys - address@hidden - http://www.xs4all.nl/~hanwen