Re: address@hidden: C++-mode: Syntax highlighting: wrong color for function identifier depending on the kind of whitespace that follows]

[Alan Mackenzie]

Good morning, Stefan!

On Tue, 14 Mar 2006, Stefan Monnier wrote:

>>> Here's the change I was talking about.  Would someone please adapt
>>> this _now_ to the current sources, and install it?  Then please rename
>>> before-font-lock-after-change-function to
>>> font-lock-extend-region-function, and rename
>>> font-lock-run-before-after-change-hook to font-lock-extend-region?

>>> Then please ack this message.

>> DONE.  In the process, I moved the processing of font-lock-lines-before
>> from font-lock-default-fontify-region into
>> font-lock-after-change-function and jit-lock-after-change, so as to be
>> able to stop them interfering with eachother.

>As discussed and agreed in this list, font-lock-lines-before should simply
>be removed since it is subsumed by the new feature.

I wasn't sure about that, but I'm certainly agreeable.

>Also, I thought we had agreed that this new feature should be in
>f-l-default-fontify-region rather than f-l-after-change-function.  I believe
>it is *wrong* for it to be in f-l-after-change-function where it is mostly
>useless:

It actually works quite well, I've tested it.  :-)  In particular, it can
extend the fontification region backwards, to some "safe" position to
start analysis, or even when refontification is wanted before the changed
line.

I'm convinced this feature needs to be in the font-lock/jit-lock
after-change-functions.  The fontification region is determined by the
_change_, not merely by the buffer contents after the change.  For
example:

1. "string \
2. over \
3. several        <========= point is at EOL 3, having just deleted a \
4. #lines."

You've just deleted a backslash on L3.  The fontification region thus
extends from L1 (where the " gets changed from string-face to
warning-face) up to L4 (where the entire line gets changed from
string-face to comment-face).

With point at L3, a static analysis in f-l-default-fontify-region could
not know that it must refontify L4.

However, I think there should be analogous features for stealth/display
fontification, something like this:

    (defvar font-lock-safe-floor-position-function nil
      "A function that determines the latest \"safe\" position for Font
    Locking analysis \(or nil).  It is given one paramter, POS.")

    (defvar font-lock-safe-ceiling-position-function nil
      " ...")

The stealth/display fontification would extend their regions using these
functions, thus fixing the problem where after-change fontification is
done right, then damaged 3 seconds later when contextual fontification
kicks in.  With these two functions, contextual fontification would be
redundant.

There is already font-lock-beginning-of-syntax-function, but it is
over-constrained.  To start syntactic analysis, you don't need to go to
the top level (i.e. outside of braces).  You merely need to go somewhere
which is "top-level" to font lock (e.g. start of a statement), or even
some point where the font-lock state is known (e.g. it's been cached). 

>you can already get the same result by using an
>(non-f-l)after-change-functions hook that sets the font-lock-multiline
>property.

OK, but for that, font-lock-multiline must first be documented.  I've
tried to figure out from the source how f-l-multiline works, but haven't
succeeded yet.  Can f-l-multiline be used to extend the fontification
region backwards?

With the f-l-multiline approach, you need somehow to ensure that the
non-f-l-after-change function gets called before f-l-after-change.  It
can be done, but is hassle.  Also, f-l-multiline is an internal Font Lock
mechanism, and major modes shouldn't have to manipulate it directly,
since it would increase the coupling between modes and Font Lock
enormously, making it expensive to change approach in the future.

#########################################################################

>> More controversially, I've explicitly documented that the region returned
>> by the f-l-extend-region may start or end in the middle of a line.

>I don't care much either way.  If someone does that, it's his problem:
>it won't hurt font-lock.el.  But it may screw up the other
>font-lock-keywords.

Why would it be a problem?  Given that f-l-extend-region-function and
font-lock-keywords both belong to the major mode, they can be made to
match eachother.

A comment in jit-lock says that if the f-l region starts in the middle of
a line, that line won't be re-displayed.  I admit I haven't tried this
out yet, but assuming it's true, would it not be easy to remove the
restriction?

Are there any other problems in starting a font-lock region in the middle
of a line?

>> I'm not sure whether that will work properly at the moment (I suspect it
>> won't), but I think it can be and should be fixed.  I'm thinking about
>> a piece of badly formatted C code something like:

>> 1. int foo (int
>> 2.          bar) {printf "Hello world"} ; int baz
>> 3.   (int omg) {

>> The "baz" on L3 hasn't a snowball's chance in Hades of getting fontified
>> with f-l-function-name-face unless the font lock region is allowed to
>> start at the "int" on L2.

>Why?  After all, when you open the file, the region will start at BOB.

When you open the file, yes.

(i) You type a space on L2, "baz" changes to variable-name-face.

(ii) You then type a space on L3, it flips back to function-name-face,
but "bar" goes to default face.  (font-lock-lines-before == 1)

(iii) Type a space on L1, "bar" is restored to variable-name-face after
~0.5 second (contextual fontification?).

(iv) reload the file, do (i), then put point over "baz" (currently
variable-name-face).  Do C-u C-x =, to display its properties.  It
changes instantly to function-name-face, which it reports in the other
window.  Hello Heisenberg!

I honestly don't think it's worthwhile to investigate each of these
happenings individually - it's amusing to watch for a few minutes, but
rapidly becomes tedious.  They all stem from a single cause, namely the
misapprehension that font lock can start analyzing code at an arbitrary BOL.
It isn't just candidates for the Obfuscated C contest that are affected.
There's an open bug report at CC Mode, thus:

  Fontification is incorrect for enums, structs, classes, etc.  of the
  following form:

  enum Enum { A, B, C }
      e;

and another about the fontification in:

    #ifndef VMS   /* VMS hates multi-line '#if's */
    # if !defined(ibm032)                    && \
         !defined(__convex__)                &&          \
         !(defined(vax) && !defined(ultrix)) &&           \
         !defined(mips)                      &&          \
         !defined(apollo)                    &&          \
         !defined(pyr)                       &&          \
         !defined(__UMAXV__)                 &&          \
         !defined(bsd43)                     &&          \
         !defined(__bsd43)                   &&          \
         !(defined(BSD) && (BSD >= 199103))  &&          \
         !defined(aux)                       &&          \
         !defined(__bsdi__)                  &&          \
         !defined(sequent)



OK, these are likely to be bugs in CC Mode's font lock patterns, but the
uncertainty caused by font-lock's instability makes them more difficult
to debug.

#########################################################################

Summary of how I think about Font Lock should be fixed:

(i) f-l-extend-region-function should be supplemented by
f-l-safe-\(floor\|ceiling\)-position-function; if non-nil, these
functions give guaranteed good boundaries for fontification.  They would
render contextual fontification redundant.

(ii) Code expanding the font lock region to whole lines should be
removed (except as in (iii)).  font-lock-multiline would become
redundant.

(iii) When any of these three variables is unset, the pertinent region
boundary should be chosen as it currently is done.

(iv) The interface between Font Lock and the display code should be
amended so that a font lock region need not start at column 0.

(v) Any remaining restrictions on multi-line patterns should be removed.

>        Stefan

-- 
Alan.