Re: address@hidden: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Stefan Monnier]

>>> (i) font-lock-fontify-region would no longer be fontifying the region
>>> specified by its paramters, but a different (possibly larger) one.

>> That's already the case since font-lock-fontify-region will typically not
>> fontify from BEG to END from the beginning of line before BEG to the end of
>> line after END.  Then that can be extended yet further because of
>> font-lock-multiline.  Nobody has ever complained about this.

>> So why is it a problem?

> "Nobody complaining" has never been good evidence for something being OK.
[ Follows some exaggeration which I find tends to weaken your point, so
  I prefer to cut it. ]

But the fact is:
- font-lock-fontify-region doesn't have a docstring a doesn't seem
  documented in the Emacs manual either.  So when you say "functions MUST do
  what they say they do", it's not clear to me that font-lock-fontify-region
  says anything about what it does.
- font-lock-fontify-region AFAICT has *never* limited its scope to
  BEG...END.  So anybody expecting font-lock-fontify-region to only fontify
  from BEG to END is deluding himself.

Basically, the way it has been defined and used until now is the following:
fontify the smallest region that includes BEG..END and which doesn't
start or end in the middle of a font-lock-keyword.

For font-lock-keywords which can span several lines, this implies extending
the region by the corresponding number of lines.  Now the region matched by
a given font-lock-keyword can't be automatically determined by
font-lock-fontify-region, so the heuristic used is to extend the region to
a whole number of lines and if that's not enough some other piece of code
(e.g. user-provided code) has to tell it via the font-lock-multiline
property (or as I suggest, via a font-lock-extend-region-functions hook).

Now let's say you have a multiline element which your font-lock-keyword is
somehow able to (re)highlight line-by-line, then you indeed don't need to
extend the region in font-lock-fontify-region.

Of course it gets trickier when your font-lock-keyword can (re)highlight
your multiline element line-by-line but only if the first line of the
element has already been fontified (where it recognizes the element and
somehow marks the text so that fontification of further lines is done
properly).  And even worse if the fontification of the first line only
recognizes the whole element depending on some text further down
(e.g. a terminator), since then the first (and succeeding) line(s) need(s)
to be somehow rehighlighted when the terminator is added/removed.
For such a situation, I currently don't know of a "canonical" way to get
font/jit-lock to do the right thing, other than to simply force the whole
multiline element to always be rehighlighted as a whole (i.e. defeating your
efforts to make your font-lock-keyword able to (re)highlight on
a line-by-line basis).

If that's the case you're trying to solve, then an after-change-function
which marks the whole multiline entity for refontification whenever the
terminator is added/removed won't cut it: if the first line is outside of
the window, jit-lock won't refontify it.

Maybe what will cut it is: using some kind of after/before-change-function
(which doesn't have to be in font-lock-after-change-function) detect when
a terminator is added/removed and in that case add a font-lock-multiline
property on the whole element (which will cause the whole element to be
(re)fontified not matter what jit-lock thinks, but the whole-element
fontification will only happen this one time since the font-lock-multiline
property is removed afterwards).

Another way: move the special treatment of the first line from
font-lock-keywords to font-lock-syntactic-keywords.  Then the
after/before-change-function only needs to move
jit-lock-context-unfontify-pos.

Yet another way: again, move the special treatment of the first line from
font-lock-keywords to font-lock-syntactic-keywords, and have it place
a jit-lock-defer-multiline property on the whole element.  But if you need
the terminator to determine whether and where is the whole element, you
clearly can't use that since you wouldn't know where/when to put that
property such that adding a terminator causes the
appropriate refontification.


        Stefan