Re: Token push-back?

[Hans Aberg]

On 25 Apr 2007, at 17:39, Evan Lavelle wrote:

> I'm just about to write a yylex wrapper to let me push back a token  
> so that Bison can re-read it. Before I do this, can anyone tell me  
> if there's a better way to handle this?
>
> My problem is that I need to write a partial parser, which ignores  
> large chunks of a language. The chunks are delimited by known  
> tokens, so I need to do something like
>
> ignore_block
>    : STARTBLOCK {
>      // check yylex input for ENDBLOCK, push it back
>      } ENDBLOCK ;
>
> ie. something like a flex exclusive start state, but initiated from  
> Bison.

You can still alter the lexer start conditions from the Bison  
generated parser. Make a variable that Bison can alter, and which the  
lexer checks every time it is being called (by some code right before  
the rules in the .l file). The biggest problem is that the parser may  
or may not need a token for lookup. So it is best to put lexer state  
changes before a token in the parser .y grammar. Then check that it  
works in practice (that is, the context changes before any new tokens  
are being read), as the LALR(1) parsing algorithm that Bison uses is  
somewhat unpredictable.

The problem you mention, making a partial parser, has been discussed  
in this list before, I think a couple of times. I do not immediately  
recall when. But perhaps you mighth find it by a search in the archives.

  Hans Aberg