Re: Eliminate MARKUP_IDENTIFIER and MARKUPLIST_IDENTIFIER (issue 6561053)

[dak]

On 2012/09/29 13:37:28, janek wrote:
> On Sat, Sep 29, 2012 at 3:10 PM,  <mailto:address@hidden> wrote:
> > Embarrassingly, not really.  The point of this change is to move
forward
> > in removing the special-casing of various *_IDENTIFIER token types.
The
> > target is to arrive at only a single SCM_IDENTIFIER type.  Once that
is
> > achieved, it is no longer necessary to evaluate identifiers in the
> > lexer, removing a source for awkward timing problems like

> So the idea is to make the grammar simpler (use less different types)
> and have things happen on higher level?  Sounds reasonable, i guess...

Actually, it is not really "simplifying" the grammar.  At the current
point of time, the lexer has to make decisions about the type of
identifiers (and functions) at a point of time where those decisions are
actually premature.  As a result of those decisions, different tokens
are generated for identifiers and functions.  I am moving towards
generating only one kind of token.  The same kind of decisions as before
still have to be made, and since the lexer no longer makes them, the
decisions are instead taken in the actions of the parser.  The results
of those decisions are only apparent in the parser rules when those
decisions lead to "synthetic" tokens (similar to the previous "real"
ones) being generated in the parser actions.

So the distinctions that were previously made are still there, they are
just done at a later point of time, and result in different actions
through different mechanisms.  Of course, once the lexer only delivers
identifiers in a single way, there are some simplifications possible,
but the overall architecture is still a worse fit to the Bison/Flex
pairing of parser/scanner generator than the state from two years ago,
as that pairing is based on lookahead principles with rigid tokenization
rules that are not really a perfect fit for LilyPond's sort of dynamic
typing.

So overall, the parser is not getting simpler.  I can straighten out
some stuff when passing certain milestones, "boiling down" complexity
again, but since I am moving functionality from where it is "intended to
be" from the view of the basic Bison/Flex architecture, the best it can
be is a consistent subsystem not looking more complex than the task it
is taking over.

http://codereview.appspot.com/6561053/