C++, locations

[Hans Aberg]

I am experimenting with Flex under C++, trying to get it to work with Bison:

- One problem is that I would want yylex to have the semantic value as an
argument. Thus, something like:

  class FlexLexer { // Or: class lexer inside namespace
  public:
    // Or "lex" instead of "yylex":
#if !YYLSP_NEEDED
    token_type yylex(semantic_type&);
#else
    token_type yylex(semantic_type&, location_type&);
#endif
    ...
  };

 (I do not see any point in Flex's virtual stuff, as that is usually used
when one has the need for a pointer to vary dynamically at runtime. This
might be that one for some reason wants to be able to switch between
different lexers at a runtime. The lexers I have used are though always
fixed, tied to the parser.)

Once I try the more explicit typing above, then the problem arises that the
file FlexLexer.h is inserted into the lexer code output file in a manner so
that it is not possible to read something before it is read. Thus, one
cannot leave
  token_type   semantic_type   location_type
undefined, as it the will generate compiler errors, and in addition, header
macros like YYLSP_NEEDED cannot be set before the file is read FlexLexer.h
from the output file.

This problem also happens with Bison: Then I have to tweak Bison so that I
can use multiple %{...%} in the .y file (in current Bison that is tied to
%union, which is not a very useful feature under C++), putting one part
into the .h header and the other in the source.

- Another problem is with locations, which is evidently tied to Bison
compilation under C. -- So I can't experiment with it that under C++.

This seems to be needed in order to generate better error messages: If one
writes error messages where only the line number is written out, it is
pretty tough to figure out what causes the error.

So I want that locations feature to not be tied to anything specific like
Bison compatibility. Possibly, if there is an %option bison-bridge, then it
should cause locations to be written on the old Bison C++ format.

So I want a special:
  %option locations.

-- Note though that the Bison M4 approach makes it easy to write new
skeleton files. Thus, I do not bother too much over which formats Flex will
use, if only the format itself is good and useful, as I will write my own
Bison skeleton file.

As for what this option should produce, then it would suffice for me that
one gets to know the latest identified token in C/C++ style semi-open
[begin, end) intervals. The data needed are pairs (l, m), where l = line
number and m = line character count, but also stream position numbers. The
latter are needed in for example my IDE (Integrated Development
Environment), which would use stream position numbers in order to highlight
the token. -- I therefore think this might be used in other similar
situations, when one needs to pin out the token location independent of
line number count.

There was a long thread about locations recently: I find it simply too
distracting to develop a new language and on the same time trying to figure
out locations, which is a side issue from a that point of view. It is also
not possible to figure out the stream position directly from the stream, as
Flex generates a buffered lexer. On the other hand, good error messages
really help, even at the language development stage.

So it would be great if Flex could support such locations to those that may
need it.

  Hans Aberg