font-lock-mode

[Luc Teirlinck]

This bug report will be sent to the Free Software Foundation,
 not to your local site managers!!
Please write in English, because the Emacs maintainers do not have
translators to read other languages for them.

In GNU Emacs 20.6.1 (sparc-sun-solaris2.5.1, X toolkit)
 of Thu Mar 23 2000 on manatee
configured using `configure  --with-gcc=no --prefix=/usr/local/emacs20'

Please describe exactly what actions triggered the bug
and the precise symptoms of the bug:

font-lock-fontify-anchored-keywords needs to be enclosed in a
save-excursion.  (In Emacs20.6, it is not.  I do not know about 20.7
or 21 as planned.)  Although I am less sure of this,
font-lock-fontify-keywords-region probably could use a save-excursion
too.

The fact that font-lock-fontify-anchored-keywords is not enclosed in a
save-excursion causes some very obnoxious bugs in a variety of
situations.  I could give concrete examples, and will do so if asked,
but they are rather complex to describe and it is obvious from the
code that the save-excursion is absolutely necessary.  (See below.) If
this problem is not already fixed in 21, I believe it should be.  I
could write pages about why it is important to do so, but since fixing
the problem is trivial, I do not believe this is necessary.  Although
I do not know any bugs resulting from
font-lock-fontify-keywords-region failing to be enclosed in a
save-excursion, and although I am less sure of this, it is probably
safer and better to enclose this function in a save-excursion too.
Both functions are supposed to fontify, and not to move point or mark.
They can move point temporarily, but they have to put it back where
they found it ,or, at least in the case of the first function, major
problems can result.  It seems obvious that if functions are supposed
to do one specific thing (fontify) and one allows them to have
important, totally unrelated and unpredictable side effects such as
leaving point in a different place, one is just begging for bugs.
Here is the code in question.  I start my personal comments with !! in
order to distinguish them from comments present in the source.
Note that in most situations my !!-remarks refer to the line ABOVE, 
unlike the usual ;-remarks.  (This is to avoid intermingling of both.)

(defsubst font-lock-fontify-anchored-keywords (keywords limit)
  "Fontify according to KEYWORDS until LIMIT.
KEYWORDS should be of the form MATCH-ANCHORED, see `font-lock-keywords',
LIMIT can be modified by the value of its PRE-MATCH-FORM."
  (let ((matcher (nth 0 keywords)) (lowdarks (nthcdr 3 keywords)) highlights
        ;; Evaluate PRE-MATCH-FORM.
        (pre-match-value (eval (nth 1 keywords))))
    ;; Set LIMIT to value of PRE-MATCH-FORM or the end of line.

!!Note that end-of-line might be larger than the original "limit",
!!which will carry point outside the region considered by this
!!function's main caller (the one below), which causes that caller 
!!to malfunction.  That point temporarily moves outside the region is
!!no problem whatsoever,only that it stays there after the function returns.

    (if (and (numberp pre-match-value) (> pre-match-value (point)))
        (setq limit pre-match-value)
      (save-excursion (end-of-line) (setq limit (point))))

!!this save-excursion puts point back in a safer place,but this is not
!!the end of the story

    (save-match-data
      ;; Find an occurrence of `matcher' before `limit'.
      (while (if (stringp matcher)
                 (re-search-forward matcher limit t)

!!here point could move in harm's way again and sometimes it stays there

               (funcall matcher limit))
        ;; Apply each highlight to this instance of `matcher'.
        (setq highlights lowdarks)
        (while highlights
          (font-lock-apply-highlight (car highlights))
          (setq highlights (cdr highlights)))))
    ;; Evaluate POST-MATCH-FORM.
    (eval (nth 2 keywords))))

!!Now comes the main caller of the previous function

(defun font-lock-fontify-keywords-region (start end &optional loudly)
  "Fontify according to `font-lock-keywords' between START and END.
START should be at the beginning of a line."
  (unless (eq (car font-lock-keywords) t)
    (setq font-lock-keywords (font-lock-compile-keywords font-lock-keywords)))
  (let ((case-fold-search font-lock-keywords-case-fold-search)
        (keywords (cdr font-lock-keywords))
        (bufname (buffer-name)) (count 0)
        keyword matcher highlights)
    ;;
    ;; Fontify each item in `font-lock-keywords' from `start' to `end'.
    (while keywords
      (if loudly (message "Fontifying %s... (regexps..%s)" bufname
                          (make-string (incf count) ?.)))
      ;;
      ;; Find an occurrence of `matcher' from `start' to `end'.
      (setq keyword (car keywords) matcher (car keyword))
      (goto-char start)
      (while (if (stringp matcher)
                 (re-search-forward matcher end t)

!!Note that this moves point too.  It apparently does not get put
!!back. I do not know of any bugs actually caused by this,because point does
!!not get carried outside the region determined by this function's
!!main caller.  It looks extremely iffy however.  It might easily cause
!!problems elsewhere,although again, I do not know examples.  

               (funcall matcher end))
        ;; Apply each highlight to this instance of `matcher', which may be
        ;; specific highlights or more keywords anchored to `matcher'.
        (setq highlights (cdr keyword))
        (while highlights
          (if (numberp (car (car highlights)))
              (font-lock-apply-highlight (car highlights))
            (font-lock-fontify-anchored-keywords (car highlights) end))

!!The "end" here is the "limit" of the called function's code. As pointed
!!out above, point can get carried beyond the original argument "end" 
!!given to the caller.


          (setq highlights (cdr highlights))))
      (setq keywords (cdr keywords)))))

!!let us follow the second function through one more iteration.

(unless (eq (car font-lock-keywords) t)
    (setq font-lock-keywords (font-lock-compile-keywords font-lock-keywords)))
  (let ((case-fold-search font-lock-keywords-case-fold-search)
        (keywords (cdr font-lock-keywords))
        (bufname (buffer-name)) (count 0)
        keyword matcher highlights)
    ;;
    ;; Fontify each item in `font-lock-keywords' from `start' to `end'.
    (while keywords
      (if loudly (message "Fontifying %s... (regexps..%s)" bufname
                          (make-string (incf count) ?.)))
      ;;
      ;; Find an occurrence of `matcher' from `start' to `end'.
      (setq keyword (car keywords) matcher (car keyword))
      (goto-char start)
      (while (if (stringp matcher)
                 (re-search-forward matcher end t)

!!Point now could be beyond "end".  Result:
!!ERROR: Invalid search bound (wrong side of point).


;;; End of Keyword regexp fontification functions.

Recent input:
M-x r - e - b return

Recent messages:
Loading bytecomp...done
Loading byte-opt...
Loading byte-opt...done
Loading personal-emacs-extensions...done
For information about the GNU Project and its goals, type C-h C-p.
Fontifying *scratch*...
Fontifying *scratch*... (regexps............)
For information about the GNU Project and its goals, type C-h C-p.
Loading emacsbug...
Loading emacsbug...done