Re: Dealing with errors generated while lexing

[Akim Demaille]

Le 6 déc. 2012 à 13:05, Nikolai Weibull <address@hidden> a écrit :

> Hi!
> 
> Is there a simple way for yylex to tell the parser to YYABORT?

Hi Nikolai,

> I’ve tried to figure out how to deal with errors generated while
> lexing.  Say that I hit an out-of-memory condition while lexing and
> want to abort the parse.  I see no easy, general way of reaching a
> point where I can YYABORT.  It’s manageable for tokens that have a
> value, as you can simply check for an error value, for example, NULL,
> but I can’t seem to come up with a good solution for non-value tokens.
> I would like to be able to return an error token from yylex, but
> there doesn’t seem to be any provisions for doing so.  I guess one
> could return a token that’s not allowed anywhere, but that would
> generate a syntax error, which isn’t the kind of handling that I want.
> One could further try to handle this error token, but then one would
> have to add a rule for that case in a lot of places.  You could add a
> value to the token as well and check for an error value, but it won’t
> work for rules such as
> 
> a: TOKEN b;
> 
> as you won’t be able to check if TOKEN has the error value until b has
> been seen and the action has been entered.
> 
> I was also thinking of using setjmp in an %initial-action and then
> longjmping to it if an error occured and from there YYABORTing, but if
> I understand my ISO C, yychar, yylen, and other auto variables that
> the yyreturn code depends upon will have indeterminate values, meaning
> that cleanup might not be performed correctly.  I noticed that the glr
> parser uses this kind of code to get out of out-of-memory conditions,
> but as far as I can tell, its code is similarly susceptible to using
> indeterminate values for these variables.  Am I wrong?

In the case of the C++ parsers, we provide support for exceptions
thrown from the scanner, but there is no provision for this in C, you
are right.

I don't think you have problems with yylen and the stack in general,
and yyresult would be set by YYABORT anyway, so it should not be a
problem either.  For the lookahead, I don't know for sure.

Maybe you could work out a sufficient, self contained, test case that
could enter the test suite, and propose a patch so that Bison does
what you expect?