"Font-lock is limited to text matching" is a myth

[Daniel Colascione]

On Monday 10 August 2009, Xah Lee wrote:
> emacs's syntax coloring is based on text pattern matching. It might work
> faster or more smooth right now, but from technology point of view, its one
> generation behind.

This is a common misconception I'd like to dispel. font-lock is *NOT* limited 
to using regular expressions. Yes, regular expressions are its default mode of 
operation, and for most languages, they are sufficient. But font-lock has the 
ability to mix and match several different kinds of fontification. Let's not 
reinvent the wheel.

There are two approaches to go beyond regular expression matching in font-lock: 
first, one can use a function as a font-lock matcher instead of a regular 
expression. That function can be arbitrarily complex, and can look at as much 
of the buffer as is necessary to fontify correctly. When this function locates 
a pattern to match, it returns control to font-lock, which fontifies it like 
any other interesting tidbit it found.

If that's not good enough, you can also have your font-lock "matcher" actually 
perform the fontification itself. cc-mode, espresso-mode, and nxml-mode use 
this technique. The basic idea is to find interesting patterns in the region of 
interest, then call font-lock-apply-highlight on them to perform the actual 
fontification. Because the "matcher" function is still using the font-lock 
infrastructure, font-lock will respect the fontification and behave correctly 
in other respects.

Using one of these two techniques allows a mode to use arbitrarily-complex 
fontification while still integrating with user-supplied keywords (e.g., from 
font-lock-add-keywords), various font-lock enhancement modes like jit-lock, and 
all other normal Emacs infrastructure.

It's *really* not necessary to bypass font-lock to get fontification to be as 
complex as you want. Just look at cc-mode.

> it'd be nice if emacs provides syntax parsing infrastructure.... nmxl, js2,
> cedet, can be leads in this direction and see how things work out.

It already does. It's called font-lock. If you want some kind of infrastructure 
that uses a grammar to perform fontification, I believe CEDET, which will be 
included in Emacs, can be wired up that way [I think? I'm not quite sure 
whether Semantic's tags are generic enough for fontification].

But using font-lock is not "a generation behind". Really, most editors' 
hard-coded highlighting features are a generation behind font-lock!