Tokenizing

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Do you plan to reparse continuously to keep the token structure up to
date all the time?

Given that reparsing is likely to take time, it may not be the best approach.
It could sometimes make Emacs become nonresponsive while it is reparsing
a substantial amount of text.

Ideally it should be like Font Lock -- reparsing parts of the buffer
when time is available.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.