Re: flex snapshot/beta

[Hans Aberg]

At 14:52 -0800 2002/03/06, John W. Millaway wrote:
>> If you have not already made it that way, I think one should be able to
>> choose if one wants to have EOL to be \n (UNIX), \r (MacOS), \r\n (DOS,
>> MSOS) or a combination of them (like all three). If the line counting
>> feature is on and \r\n is admitted as EOL, it should only increment line
>> numbers by one, not two.
>
>The change mentioned is only for flex's INTERNAL scanner (not the generated
>scanner's input.) It only affect EOLs in "foo.l". The EOL test used in the
>GENERATED scanner is unchanged, and is merely a check for '\n'.

Sorry, I did not notice that.

>But you're right: If we're going to have the optional automatic EOL test,
>(given the performance penalty, yadda yadda yadda) then we should allow the
>test to be configurable. At least, one should be able to specify one of the
>three common cases you listed above.
>
>The only tricky case is when the user wants flex to detect "\r\n", but then
>fouls it all up by writing patterns that split the token, like this:
>
>  /* User tells flex that \r\n is the EOL sequence. */
>%%
>  /* But the grammar may explicitly want CR and LF
>     as separate tokens. */
>\r   { return CR; } /* No EOL here, says flex. */
>\n   { return LF;}  /* No EOL here either!. */
>
>%%
>
>Now, I'm lying a little, because the input is buffered, so flex could
>technically look behind in the buffer for the start of the eol sequence.

Could you not simply introduce a new variable
  char last_char;
that records the last char in a scan? When the next scan begins, if
last_char == '\r' and the current char is '\n', one knows that the line
number should be incremented (or has been incremented by the '\r'). (When
distinguishing between pre-/post-/in-fix operators of the same name, one
may need a last_token variable. So this is where I stole the idea.)

  Hans Aberg