Inserting extra tokens

[Erik Sandberg]

Hi,

I am working with the parser of GNU LilyPond, and I want to improve the way 
function invocations are parsed in the language. The language uses a syntax 
for functions which is somewhat similar to LaTeX's syntax for macros. E.g., 
if \foo is declared as a binary function, then \foo a b calls the function 
with parameters a and b.

I'm looking for a way to express this syntax in bison, in a generic way 
(currently only a limited number of function arities are possible). I think I 
have found a solution that would work, but I'd first like to hear if you 
think it is improper use of bison, and if you know better ways.

My proposed solution is as follows:
1. The function's arity is encoded in the \foo token, so both the parser and 
lexer knows the arity.
2. When a function argument has been shifted, the parser artificially inserts 
a special token as the next token. The token is a COMMA or SEMICOLON 
depending on whether the shifted argument was the last argument.
3. The argument list uses a grammar similar to
arglist: argument COMMA arglist | argument SEMICOLON ;
(an argument can be a complex expression)

A problem in (2) is _how_ to insert the extra token correctly. I first tried 
to put flex in a state where it sends the desired token without reading 
anything. This was however not sufficient: In some cases, the parser has 
already read the next token for the lookahead when an argument is shifted.

I am thinking about the following workaround:
- if the lookahead is empty, just make the lexer spit out a COMMA/SEMICOLON.
- if the lookahead is nonempty, change it to a COMMA/SEMICOLON and make the 
lexer spit out the previous content of the lookahead.

Would this work? Is there a better way?

-- 
Erik