Re: Last steps for pretesting (font-lock-extend-region-function)

[Alan Mackenzie]

Hi, Stefan!

On Thu, 20 Apr 2006, Stefan Monnier wrote:

>>> So your code needs extend-region both in a-c-f and in f-l-d-f-r ?

>> Yes.  The region needs extending in two distinct circumstances:
>> (i) In after-change, to determine the entire region which needs
>>   refontifying;
>> (ii) In jit-lock-fontify-now, to ensure that each chunk starts and stops
>>   at a "safe place".
>> These are logically disjoint.  But it seems that the major mode code to
>> do each of them is so similar, that we might as well just have a single
>> function, font-lock-extend-region-function, called in two different ways.

>IIRC part (i) needs more than an after-change-function: it also needs
>a before-change-function.

That would depend on the major mode, I think.  AWK mode needs a
before-change function.  The CC Mode replacement for (setq
font-lock-lines-before 2) will not need a b-c function.

However, using the f-l-multiline property approach instead will also need
a before-change function for exactly the same reasons - or else it will
require the major mode to maintain the f-l-multiline property
permanently.  That would be a major overhead.

>And (again IIRC) it's the part that can already be solved by placing
>(from one of the font-lock-keywords rules) a font-lock-multiline
>property.

I'm not sure.  Can a font-lock-keyword rule get access to after-change's
OLD-LEN in an reasonable fashion?

If a change consists of yanking 20k of text into a buffer, the Major Mode
must not splat font-lock-multiline onto 20k + 20 bytes, since this would
defeat the purpose of JIT lock (fontifying a bit at a time).  The Major
Mode would have to apply f-l-m separately to the head and tail of the
(large) region separately, but in the normal case (small change region)
would have to combine them as a single region.  It would need logic do
distinguish these two cases.  This is inelegant.

>Basically it's the part that makes sure that if a multiline
>element loses its multiline status, the whole (previously atomic) text
>gets properly refreshed.

How is this to be done with the multiline property?

>I find it very hard to believe that b-f-c + a-f-c ends up being as
>efficient/elegant/robust as placing a font-lock-multiline property.  In
>all likely hood, adding the font-lock-multiline property is a matter of
>adding a single line of code at the right place (the place where you
>fontify the corresponding multiline entity).

Stefan, the font-lock-multiline property is still undocumented.  That
makes it very frustrating, very difficult to evaluate the paragraph
you've just written, very difficult to criticise it constructively (I
think I've perhaps already done enough of the other sort of criticism
;-).  

Exactly where the "right place" is to add that single line of code is
completely opaque to me at the moment.  I could probably work it out with
several hours study of the Font Lock source and documentation, but I
don't want to do that at the moment - I've got enough on my plate.

Could you please document the font-lock-multiline mechanism immediately,
including details on how best to access the after-change-functions's
OLD-LEN, and details of how and where to hook in the "(put-text-property a
b 'font-lock-multiline t)" call.  Either that, or tell me where to find
an example of this text property being used, preferably one which uses
OLD-LEN.

Or, perhaps you could recode the following AWK Mode fragment into an
equivalent which uses f-l-multiline instead of f-l-expand-region
function.  PLEASE!

#########################################################################
(defvar c-awk-old-EOLL 0)
(make-variable-buffer-local 'c-awk-old-EOLL)
;; End of logical line following the region which is about to be changed.  Set
;; in c-awk-before-change and used in c-awk-font-lock-extend-region.

(defun c-awk-before-change (beg end)
;; This function is called exclusively from the before-change-functions hook.
;; It does two things: Finds the end of the (logical) line on which END lies,
;; and clears c-awk-NL-prop text properties from this point onwards.
  (save-restriction
    (save-excursion
      (setq c-awk-old-EOLL (c-awk-end-of-logical-line end))
      (c-save-buffer-state nil
       (c-awk-clear-NL-props end (point-max))))))
(add-hook 'before-change-functions c-awk-before-change nil t)

(defun c-awk-end-of-change-region (beg end old-len)
  ;; Find the end of the region which needs to be font-locked after a change.
  ;; This is the end of the logical line on which the change happened, either
  ;; as it was before the change, or as it is now, whichever is later.
  ;; N.B. point is left undefined.
  (max (+ (- c-awk-old-EOLL old-len) (- end beg))
       (c-awk-end-of-logical-line end)))

(defun c-awk-font-lock-extend-region (beg end old-len)
  (cons (c-awk-beginning-of-logical-line beg)
        (if old-len
            ;; From an after-change-function
            (c-awk-end-of-change-region beg end old-len)
          ;; From a fontifying routine
          (c-awk-end-of-logical-line end))))
(setq font-lock-extend-region-function 'c-awk-font-lock-extend-region)
#########################################################################

>Here is the problems I see with your proposal:
>- an `extend-region' hook in font-lock-fontify-region is needed

It is indeed.  It is a simple patch, and I am ready and willing to write
it, both for font-core.el and modes.texi.  I do first ask that it's
accepted in principle, though.

>- the font-lock-multiline property should be enough in all cases to make
>  it unnecessary to use an after-change-function hook.

f-l-extend-region-function also makes a (separate) after-change-f hook
unnecessary.

>- adding an after-change-function hook additionally to
>  a font-lock-fontify-region hook means making font-lock that much more
>  complex to use and maintain, especially since that after-change-function
>  basically can't be used without also adding a before-change-function.

I disagree with everything in that paragraph.  :-)

>- an after-change-function is expensive, and even more when it comes bundled
>  with a before-change-function, and even more so when compared to
>  code that's only run during font-locking.

Huh?  Font Locking _is_ expensive.  A before-change function need not be
expensive (e.g.  c-awk-before-change, see above), and calling it is not
expensive.  I suspect that using the f-l-multiline property will
absolutely mandate an extra after-change function - how else can it
access OLD-LEN?

Stefan, please show me the code!  (Or the documentation!).  How does a
hacker use the font-lock-multiline property?  In detail.

-- 
Alan.