Re: reentrant flex, bison glue

[Akim Demaille]

>>>>> "John" == John W Millaway <address@hidden> writes:

>> The thing is, you are putting a contract onto us.  You are imposing
>> an interface onto Bison, where there has never been one before.

John> Aha! So that's what eating you. Why didn't you voice this
John> concern in the first place? 

Because it's not only this, it's also something I find it hard to
explain, which is related to the motivation.  All my Autoconf years
made me somewhat suspicious to features which are not needed: they
always end up into maintenance burden.  Because there is always
something you didn't think about.

John> Here is my defense:

:)

John> 1. The bison manual is a contract. Bison is expected to behave
John> as described in the manual.

John> 2. The bison manual details the C API. Since bison is mature,
John> nobody expects the C API to suddenly change.

You missing something: you are talking of bison producing LALR C
parsers only.  I'm talking about Bison.  We already have a functional
C++ Bison.  We should have DLR, and maybe other techniques.

Also, we are bound to maintain the old interface, granted, but I will
never forbid me/s.o. to introduce another interface if it turns out to
be more intelligent etc.

This interface is 25 years old.  Committing it today, 25 years
afterwards, doesn't sound a promising approach :) (And, BTW, the
interface has never ever referred yylval/yylloc: you are adding
this!).


But again, my point is more like ``pay attention that a feature has a
cost and an added value: be sure to have an interesting ratio,
otherwise you pay a high value for a limited additional service''.




>> In particular because I have still not seen what exact gain the
>> users can be expect from this feature.

John> The gain is usability and consistency. It is a pain in the ass
John> to glue pure bison to reentrant flex, unless you already know
John> how to do it. Even then, it requires a ride through the manuals,
John> and you wonder why the tools didn't just do it _for_ you. The
John> glue provides consistency with the rest of the flex API.

My opinion is that Bison is the one that can make an effort for Flex,
not the converse.  Bison is on top of the latter, not the converse.
For instance, it's been a while I've been thinking having token rules
in the .y, and produce the .l from it.


>> What Flex functions other than yylex could have (could be willing
>> to have) an access to yylloc and yylval?

John> Not Flex functions, user-defined functions. Flex may not care
John> about yylval, but the user does. There is no reason for flex to
John> impose restrictions on WHERE a user is allowed to access
John> user-accessible variables. They are global in non-reentrant
John> flex, and I want to preserve that "globalness" in reentrant
John> flex.  Here's an example where the data needs to be accessible
John> from outside yylex:

John> // called by bison
John> int wrapper_yylex() {
John> 
John>     // call the real yylex
John>     int tok = yylex();
John> 
John>     // post-process the token
John>     switch(tok){
John>     case FOO:
John>     case BAR:
John>     case FUM:
John>         if( yyleng < 1234 )
John>             yylval->str[yyleng-1] = '\0' ;
John>          tok = FOO;
John>          break;
John>     }
John>     return tok;
John> }

Err...  You really want this to work?  That explains everything!  I
would never have wanted to have secrete arguments!  In the case of a
pure parser, I *wanted* to see the additional arguments passed to both
yylex:

// called by bison
int wrapper_yylex(YYSTYPE *yylval, YYLTYPE *yylloc, 
                  comment_accumulator_t *yylcmt) 
{

    // call the real yylex
    int tok = yylex(yylval, yylloc, yylcmt);

    // post-process the token
    switch(tok){
    case FOO:
    case BAR:
    case FUM:
        if( yyleng < 1234 )
            yylval->str[yyleng-1] = '\0' ;
         tok = FOO;
         break;
    }
    return tok;
}

Basically, I guess the reason I could not understand you is that I
don't expect Bison and Flex to try to be smart with my code.  I want
them to expand the rules, period.  I'm the chief for the rest of the
code.

I suppose it means the user must not have code using the names yylloc
and yylval but as local variables?  I also guess you can't use this
feature when you renamed yy to foo? (I mean, using Bison's
--name-prefix, not Flex's equivalent option).