common per-token info

[John W. Millaway]

Akim: Just for the sake of example, can you give us a scenario for which you
need to track the "width" of each token, (where width != yyleng) ?

Everyone: What are some example parsers/grammars that need per-token
information not covered by the list [file,line,column, USER_DEFINED_MEMBERS].
Let's assume that we can implement optional file, line, and column tracking
mechansims that are reasonably configurable (zero-based or one-based, handles
"\r\n", etc.), and that do not tarnish the holy grail of DFA speed. What will
we miss? Is this too much information per-token? Do most parsers unwrap tokens,
then rewrap them in user-defined objects at the parser level anyway (discarding
the information gathered by the scanner)?
 
What I'm looking to define is a practical common base of per-token information
that is expected by the various grammars and parser generators out there.

__________________________________________________
Do You Yahoo!?
Send FREE video emails in Yahoo! Mail!
http://promo.yahoo.com/videomail/