Flex & Unicode

[Frans Englich]

Hello,

I need a unicode-compatible Flex, and from visiting Google & reading the 
manual I understand that Flex doesn't have that out of the box, but one have 
to apply The unicode patch.

So I did that. When compiling the produced scanner(after running flex with 
-U), I get the following error for the line "Mylval.sval = new 
StringImpl(&(Mytext[0]))":

MyLexer.l:72: error: call of overloaded `StringImpl(YY_CHAR*)' is
   ambiguous
StringImpl.h:39: error: candidates
   are: StringImpl::StringImpl(const StringImpl&) <near
   match>


So I googled, and found this thread from January 2002:

http://lists.gnu.org/archive/html/help-flex/2002-01/msg00126.html

where Mark Weaver writes:

<quote>
The unicode flex defines "YY_CHAR" as the type used for compiling the
scanner.  The re-entrant flex defines a new state structure for each scanner
(PREFIX_flex_state).  To make the two work together, I've had to define the
character type PREFIX_CHAR in the generated header file for a re-entrant
scanner.  There is an icky ordering issues in the way that the re-entrant
header is included which means that YY_CHAR is needed before including
YY_FLEX_STATE (specifically, that YY_BUFFER_STATE needs to be defined).  To
get around this, I protect the definition of PREFIX_CHAR with the macro
PREFIX_CHAR_DEFINED, and reproduce the definition of YY_CHAR in the scanner
file.  It doesn't look amazingly pretty, but it does work, and this only
applies when compiling the generated scanner; the unpretty bit is the extra
macro.  I'd be interested to know if anyone can think of a way around this.
(The YY_BUFFER_STATE structure of course uses YY_CHAR itself - hence the
slightly icky solution).
</quote>

I wonder: how do I get a functioning Flex scanner that supports Unicode? 
Should I go to work with Mark's comment and look closer at that, or is there 
a simpler answer? Should I define YY_CHAR or some other define, and to what,  
and where?


I also re-ask what Guillaume Morin started that thread back in 2002 with: any 
chances that Flex acquires Unicode support? What is the reason to it haven't 
been incorporated, is there technical problems? It would in either case be 
great to be able to assume Unicode here in 2005, without having to go 
patching sources.


Cheers,

                Frans