Considering other lexer implementations to better flex

[Clint Olsen]

I was having a discussion with John Millaway about this, so I thought I
would post it to the help-flex forum.

---------------

At one time I was a big flex user.  However, when I was looking for a
reentrant lexer, flex wasn't there (yet).  Re2c isn't as featureful as flex
(e.g. no support for exclusive start states), but it's extremely flexible
since you control the function that does the scanning.  All it does is
match regular expressions - little else.

The result of my benchmarks about a year ago were posted on comp.compilers.
The original re2c article published by Peter Bumbulis claimed faster lexers
than even flex.  I didn't see a huge speed difference in my testing, _but_
I was able to easily incorporate line _and_ column tracking and at least
match flex's speed (only tracking line number), and the resultant lexer was
a _fraction_ of the size of flex's.  At the time I didn't know how to
(easily) track column info from flex so I didn't bother.  Personally I find
column tracking a huge productivity booster for obvious reasons.

The reason why I bring it up is that now that flex support is picking up,
it would be great if anything novel (algorithm speed and resultant lexer
size) from re2c could be incorporated into flex.

---------------

Just in case anyone's interested, here's my post to comp.compilers
addressing someone's complaint that flex was generating too large a scanner
for his particular application:

http://groups.google.com/groups?q=group:comp.compilers+olsen&hl=en&lr=&selm=01-02-123%40comp.compilers&rnum=9

-Clint