Re: Inserting extra tokens

[Erik Sandberg]

On Friday 18 August 2006 15:57, Hans Aberg wrote:
> On 18 Aug 2006, at 12:12, Erik Sandberg wrote:
> > My idea was to check dynamically whether 0 or 1 lookahead tokens
> > has been
> > read, and take different actions depending on this (either queue a
> > token in
> > the lexer, or manipulate the current lookahead token). Would this
> > fail?
>
> The parser must know when it needs to get a lookahead token, so it
> seems me the information should be present in some of the parser tables.
>
> > If so, do you know about any other parser generator that supports
> > insertion of
> > tokens?
>
> In general, however, people try to avoid this, for any parser. When
> one sets context switches in the parser .y file to change the
> behavior of the lexer /as in the .l file), one generally tries to put
> in some action where the lookahead problem does not matter. Then one
> checks by hand that it worked out correctly.
>
> You might alos consult the Usenet newsgroup comp.compilers, and its
> FAQ published there monthly, for more inputs.
>
> >>> 3. The argument list uses a grammar similar to
> >>> arglist: argument COMMA arglist | argument SEMICOLON ;
> >>> (an argument can be a complex expression)
> >>
> >> Now, I do not see exactly how this precedence problem relates to
> >> yours, as you have a different syntax. But if you only admit a
> >> limited number of arities, you could list them all in the .y grammar.
> >
> > We currently use something similar, I'm trying to find a more generic
> > solution.
>
> The state of the art of these parser generators is that they, with
> respect to this problem, are quite primitive. So the best solution is
> to tailor something more special.

Hi,

I found a solution to my problem: If a function takes two arguments, then the 
lexer generates three tokens for the function name:
\function
=>
FUNCTION EXPECT_ARG EXPECT_ARG
and then the grammar for the argument list looks like:
argument_list: /*nothing*/ | EXPECT_ARG argument_list argument;

which works perfectly.

Thanks for your help!

-- 
Erik