Re: How do lexical tie-ins work?

[Joel E. Denny]

On Fri, 23 Apr 2010, Zachary Carter wrote:

> I'm curious on how Bison's algorithm works wrt lexical tie-ins[1]. The
> situation appears to me as a bit of a catch-22. The parser checks the
> lookahead token to decide when to reduce (right?)

Not always.  If the parser doesn't need a lookahead because there's only 
one possible parser action, then it doesn't fetch a lookahead from the 
scanner.

> but the semantic action
> may alter the scanner so that the next token shifted is actually different
> than what the lookahead token had been. It seems counter intuitive, but is
> that really how it works?

No, the semantic action would have to modify the stored lookahead (yychar) 
not the scanner behavior (using hexflag, for example) to make that happen.

> [1]: http://www.gnu.org/software/bison/manual/bison.html#Lexical-Tie_002dins

Here's the relevant production from that example:

  expr: HEX '(' { hexflag = 1; } expr ')' { hexflag = 0; $$ = $4; }

The assumption is that, after HEX '(', the mid-rule is the only possible 
action, so hexflag = 1 is set before the next token after '(' is fetched.