Re: incorrect yychar for unambiguous GLR

[Paul Eggert]

"Joel E. Denny" <address@hidden> writes:

> So, token numbers might be greater than 2^16 (int yyrawchar),

Yes.  It's longstanding practice that token numbers must be 'int'
values.  They have to be positive, as nonpositive values represent
EOF.

> the number of tokens cannot be greater than 2^16 (short yytoken).

I wasn't aware of that restriction, but it is a limitation that we
should remove at some point.  Bison-generated parsers shouldn't have
arbitrary limits like that.

The same goes for state numbers and some other things like that.  By
tradition state numbers are 'short', but apps shouldn't (and in
practice, don't) rely on this, and we shouldn't continue to insist on
it, for grammars that need more than 32767 states.  (I think it's low
priority to remove this arbitrary limit, but I've been wrong
before....)