Re: C++ direction

[Hans Aberg]

At 14:04 -0400 2002/10/23, John Millaway wrote:
>The main point here is that the C++ scanner is problematic both
>internally and externally. Internally, because the code itself is in
>our way. The C++ scanner is essentially a code fork that is interlaced
>into the skeleton, but offers no benefits over the reentrant C
>scanner. Externally, it is problematic because it uses broken C++, the
>API is sloppy, and it doesn't support all the features that the C
>scanner does.

I think that what one wants in the first hand is being able to compile it
under the current C++ standard, so that the rule actions can be in C++.

The next thing would be a class pure parser, with a better interface, but
that can wait until one knows better how to do it. -- This would be needed
for threading and the like, but also for better interfacing with Bison,
where it ought to suffice to make new instantiations of the class.

>Many of you have cited bison's m4 approach as a path towards a
>multilingual flex. This approach is more complex than it looks. Yes,
>it is simple to carve up the skeleton into pieces, then assemble those
>pieces conditionally using m4. This is what flex does now anyway, but
>instead of m4, we use the C preprocessor and some flex-specific
>macros. But this whole approach assumes that we have created some
>internal, generic representation of the output scanner BEFOREHAND,
>which can be translated or mapped to the various m4 code fragments in
>a given output language. Currently, flex does not build such a
>representation, aside from the DFA tables.

You seem to opt for some more ambitious project:

The way Bison works now is that it just produces those thus-down automaton
tables, which are output as M4 macros. Then, in another file, one writes
that push-down automaton, together with instructions how to place the table
stuff.

Take a look at both the output that Bison hands over to M4 (I think one can
turn on some tracing option in Bison 1.75 to make the file remain on the
disk), as well some of its skeleton files. -- I work on the lalr1.cc file
for C++, and ther is one yacc.c for C.

This is in itself not very advanced, but quite neat. In what way would Flex
not be able to handle that in a straighforward manner?

> So, the REAL task here is
>not how to macro process the skeleton, but to explicitly separate
>flex's internal representation of the generated scanner from the C
>code generation.

I am not sure what you have in your mind here:

The way Bison does it is that it puts all stuff into different macros,
tables, code snippet, actions. Then that stuff is fed, together with the
skeleton file, is fed into M4. The skeleton file contains instructions that
expands those macros suitably into say C formatted tables and code segments
picked together with the actions. These are written as segments that can be
written onto different files.

Then currently, M4 outputs that to Bison, which has a special code snippet
splitting the stuff up onto different files. But, hopefully, that code
snippet will be moved over to M4.

When moving Flex towards this implementation, that should simplify, because
one does not have to know where to put the stuff when generating the
information.

I think that Bison outputs it stuff via something they call "muscles": One
puts the info into obstacks, and the muscles can output it. (Perhaps one
might be able to bypass the obstacks, and put out the macros in a more
direct manner.)

-- I can't say if this is a desirable way for Flex to go, but it can't be
very difficult, especially as it is already done in Bison.

  Hans Aberg