Re: Last steps for pretesting (font-lock-extend-region-function)

[Alan Mackenzie]

Hi, Stefan.

On Fri, 21 Apr 2006, Stefan Monnier wrote:

[ .... ]

>> What is your objection to f-l-extend-region-f in a-c-f?  Is it because it
>> might gobble too much processor time?

>That and the fact that it's not necessary, and that to use it (as seen
>in your example) you need a good bit more work/code than doing it "the
>other way" (with code solely run from font-lock-default-fontify-region).

It will need an order of magnitude less work for f-l-extend-region-f than
for f-l-multiline properties.  At least, it will for those hackers who
aren't totally familiar with the internals of font-lock-keywords, etc.

f-l-e-r-f only needs to be applied at the extremes of the region.  f-l-m
needs to be applied throughout the entire region.  I doubt very much
whether f-l-m would use less processing power than f-l-e-r-f.  Also,
f-l-extend-region-function (called from f-l-a-c-f/j-l-a-c) will work
EVERY time, because of its directness and simplicity.  Using only f-l-m
will fail in some circumstances.

In the most general situation, a regexp in f-l-keywords will be
inadequate - a defun will be needed.  This will be at least as
complicated as the one for f-l-extend-region-f.

Writing a f-l-extend-region-f does not require any detailed knowledge of
font-lock-keywords - thus the two problems of determining the region to
fontify and of specifying how to fontify it are kept separate, hence
simplified.  Abusing f-l-keywords to set exotic text properties (as
against its primary purpose, to set faces) requires contorted thinking.
To write a f-l-e-r-f, one need only read a single page of the elisp
manual.  

It's me that's going to have to implement this mechanism for CC Mode (all
modes other than AWK Mode).  It's going to be difficult enough as it is,
without the added complications of the difficult to understand structure
font-lock-keywords.

>> It needs to be in j-l-fontify-now, so that that function knows what
>> bytes to apply the 'fontified property to.

>This is not needed for correctness.  It's only a matter of avoiding
>redundant work.  This need wouldn't be new (it has existed since
>Emacs-21 becaue of font-lock-multiline) and doesn't need to be addressed
>right away.

It might cause errors, and it will certainly increase run time.  It isn't
much effort to be correct, here.  Why not do it right?  (Yes, I'm
prepared to do this patch, too.)

[ .... ]

>> I can't see how a major mode can manipulate f-l-multiline properties
>> anywhere BUT an a-c-f (or a before-c-f).

>What about the sample solution I provided: it's all manipulated from
>font-lock-keywords (i.e. from font-lock-default-fontify-region)?

It won't work in every case.  An after-change function (at the very
least) will need to be added to your solution to make it robust.

>>> Look ma!  No old-len!  Tadaaa!
>> .... but some code, somewhere, sometime needs to adjust the f-l-m
>> property, removing it from L4.  In general, it will also need to
>> _split_ the text property into L1,2,3 and L4,5,...  Otherwise, a f-l-m
>> region could gradually engulf an entire buffer, making a nonsense of
>> jit-lock.  (OK, this would be an extreme situation.  :-)

>Yes, of course.  It's done this way: font-lock-unfontify-region (called
>from font-lock-default-fontify-region) removes the font-lock-mutliline
>property on the whole fontified region just before proceeding with the
>actual fontification, so the font-lock-keywords need to re-add it at
>those places where it's still needed.

OK.

[ .... ]

>> Something, somewhere, sometime has to analyse the newly inserted code,
>> putting f-l-m properties throughout it, I think, and adjusting those
>> at the boundaries of the new region.  Is this to be done at the time
>> of the change or at the time it gets font locked?  I think it has to
>> be done at the change, otherwise the f-l-m settings will be wrong when
>> it comes time to fontify a chunk.

>Let me extend my example.  In the last episode, we stopped just before
>refontification was to take place and had seen that the f-l-m property was
>going to make the refontification apply to all 4 lines, as needed.
>The fontification will work as follows:

>5. font-lock-default-fontify-region gets asked to refontify line 3.

>6. it extends the refontified region at beg and at end because of the
>   presence of f-l-m property.  So the refontified region now spans all
>   4 lines.

>7. Prior to applying the new fontification, font-lock-unfontify-region is
>   called which removes all `face' properties and `font-lock-multiline'
>   properties on all 4 lines.

>8. the actual refontification takes place, using font-lock-keywords.
>   During this step, the rule

>     (".*\\\\\n.*"
>      (0 (progn (put-text-property (match-beginning 0) (match-end 0)
>                                   'font-lock-multiline t)
>                nil)))

>   is used to re-add a font-lock-multiline property to the first 3 lines
>   (the 4th is not part of the multiline-line any more).

>   If we had remoed the \ on line 2 instead, the refontification would have
>   added the font-lock-multiline to lines 1&2 and then 2&3 (the two
>   separated by a \n which is does not have this property).

>See, it does analyse the new text to adjust the font-lock-multiline property.

It sounds good in theory, but hasn't yet been tried in practice, AFAIK.
Or has it?  For which major mode?  I really don't want to have to be the
first person to try your approach.  I don't like it and don't think it
will work very well.

>> 1. I think font-lock-extend-region-function, called as it is from
>> j-l-after-change, is a good thing.  It is a correct way of giving a
>> major mode an opportunity to massage a fontification region.  I think
>> it should stay.  You think it should be removed.

>Yup.

>> 2. f-l-d-f-r / j-l-f-now absolutely need a
>> "font-lock-extend-chunk-function" functionality (regardless of its
>> precise name).  We agree on this. 

>Yup.

>> 3. You have convinced me that it is possible to use the f-l-m
>> properties.

>Good.

>> 4. I think that f-l-extend-region-f, called from f-l-after-change, is
>> likely to be more convenient and less error prone than the direct
>> manipulation of the f-l-multiline property by major mode code.

>You mean adding a single put-text-property call in your
>font-lock-keywords rule is more error prone than cooking up a b-c-f plus
>an a-c-f plus some way to store info between the two so that the a-c-f
>can figure out what happened, ...?

Yes, absolutely.  There are so many special cases, font-lock-keywords has
a complicated structure, it would be very easy to put the property on
".*\\\\$" rather than ".*\\\\\n", for example. 

>> 5. I suspect you think that f-l-extend-region-f is likely to consume
>> more processor time than direct manipulation of the f-l-m property.

>The problem is not what is done but when.  after-change-functions get run
>more often than font-lock-fontify-region.

No it doesn't.  font-lock-fontify-region gets run at every change (unless
jit-lock-defer-time is non-nil - it's nil by default).  f-l-f-r gets run
additionally for background fontification.

>> Is there any reason why we can't leave both methods side by side in
>> the code?

>Unnecessary complexity.  And it will mislead people into thinking that
>using a a-c-f is a good way to solve their problem.

after-change-functions is the canonical way of solving problems related
to buffer changes.  This includes Font Locking.

I say to you again - your solution is not robust.  I don't think it's
been tried at all (correct me if I'm wrong).  A variant form of
f-l-extend-region-f (the advice on the f-l a-c-functions, as implemented
for AWK Mode) has been in use since June 2003, and so far no bugs related
to it have been reported by users.  We are both aware of a refinement
which is needed, namely calling some sort of f-l-extend-region-f from
f-l-d-f-r and j-l-f-n.

So - please leave the better tested mechanism in place.

>        Stefan

-- 
Alan.

P.S.  We're boring everybody else as well as ourselves.  :-(