Argument scanning, take II

[David Kastrup]

Short of changing the argument syntax of music functions to something
less fuzzy altogether (implying GLISS), one needs a somewhat mixed
approach.  It is clear that something like
\relative c' c'  vs  \relative { c' }
offer some significant challenges with regard to argument parsing.
Something like c' can be interpreted in multiple ways, and Lilypond
can't back up when scanning (and humans are not all too good at double
takes either, so the syntax should not become too complex).

So one thing I'll be doing is to form one syntactic class "item" mostly
a superset of current "scalar".  Its principle characteristic is that it
can be parsed without requiring lookahead.  So something like <<music>>
or {music} or 42 or "string" or even \repeat volta 2 {music} is fine,
but Lilypond can't look for \addlyrics or \alternative or \cm or similar
behind it.  Things like \markup { ... } or \markuplines { ... } are also
fine as well as #something which makes it possible to specify a value as
##{ 4\cm #} or #(* 4 cm) if you really want the value for 4\cm in a
place where only items are acceptable.

Now if we have an item not requiring lookahead, the parser will be able
to back it up and use it for something else as long as it can be
interpreted in the same manner.

Now to the context-dependent interpretation of function arguments: if we
want most function arguments to be grouped into the "item" class where
the parser can consider the implication by calling the predicate of the
argument rather than looking at the predicate and parsing in a specific
manner, we have the following situation:

If we have an optional argument of class "item", the parser must be able
to look at it and decide it does not want it.  But that means that
somebody else has to take it, in the indiscriminate syntactic form of
"item".  Since music lists can accommodate quite other things rather
than items, those can't escape into the surroundings of a function call.
So there is a rule that a _skipped_ optional item will usually have to
be followed by optional or mandatory arguments that can make use of an
item.  In order not to make argument matching all too nonlinear, I want
stay with the decision that once an optional argument is skipped, all
following optional arguments get skipped as well.  So the following
mandatory mandatory argument has to accept the item regardless of its
syntactic class (it may choose to barf at its type, but it can't
delegate the barfing to the parser via the normal course of a syntax
error).

So most of the parser would not bother about distinguishing items but
just take what it gets, leaving the complaints about wrong type to the
semantics.  That would make most of the code in
Lily_lexer::try_special_identifiers trying to sort an expression into
*_IDENTIFIER classes unnecessary.

Of course, some things still have to be sorted out by the parser.
Things like pitches and durations can't easily become items because
their parsing usually requires lookahead (to see whether additional
octave marks or duration dots follow).  You can force them into items by
doing something like
pitch = #(define-scheme-function (parser location p)(ly:pitch?)p)
and then using ##{\pitch c'#}.

Not all that pretty...  Anyway, the main point was that if one wants
reasonable syntactic power in music functions without too many fine
points and breakage, a considerable amount of responsibility for
interpreting material in the typical course of operation has to be
withdrawn from the parser and put into other code.  That's a tricky
challenge when one wants to integrate it with existing uses of the
hard-coded syntax.

On the plus side, things like #"xxx" and "xxx" or #3 and 3 will be
perfectly equivalent in all conceivable circumstances, so one can just
forget about "educating users" about fine points like that.  You want to
use Scheme?  Feel free.  You don't want to?  Feel free.

-- 
David Kastrup