C++ direction

[John Millaway]

The main point here is that the C++ scanner is problematic both
internally and externally. Internally, because the code itself is in
our way. The C++ scanner is essentially a code fork that is interlaced
into the skeleton, but offers no benefits over the reentrant C
scanner. Externally, it is problematic because it uses broken C++, the
API is sloppy, and it doesn't support all the features that the C
scanner does.

Many of you have cited bison's m4 approach as a path towards a
multilingual flex. This approach is more complex than it looks. Yes,
it is simple to carve up the skeleton into pieces, then assemble those
pieces conditionally using m4. This is what flex does now anyway, but
instead of m4, we use the C preprocessor and some flex-specific
macros. But this whole approach assumes that we have created some
internal, generic representation of the output scanner BEFOREHAND,
which can be translated or mapped to the various m4 code fragments in
a given output language. Currently, flex does not build such a
representation, aside from the DFA tables. So, the REAL task here is
not how to macro process the skeleton, but to explicitly separate
flex's internal representation of the generated scanner from the C
code generation.