bug#28850: 26.0.90; Error running timer 'jit-lock-stealth-fontify': (error "Invalid search bound (wrong side of point)")

[Alan Mackenzie]

Hello again, Eli.

On Sun, Oct 15, 2017 at 19:07:50 +0300, Eli Zaretskii wrote:
> This bug is bugging me for quite some time now, and my hopes for it to
> be resolved are now gone, so I finally sat down to debug it.

> I have jit-lock-stealth turned on in my sessions, so whenever I
> restart Emacs (e.g., when I build a new binary, or after a system
> restart), and restore my session using desktop.el, Emacs starts
> fontifying in the background.  At some point, sometimes more than
> once, I get this error:

>   Error running timer 'jit-lock-stealth-fontify': (error "Invalid search 
> bound (wrong side of point)")

> Today I ran Emacs under a debugger, and caught this error.  The
> details are below, but in a nutshell, CC mode's fontification
> functions call re-search-forward with BOUND that is before point.  I
> hope the data below is enough to understand why that happens and fix
> it; if not, please tell what additional data is needed to diagnose the
> problem.

The details you've given me are enough to form a strong hypothesis.
Thanks.

> Here're the C and Lisp backtraces from the error, and some relevant
> data that explains why the error happened:

[ .... ]

>   Lisp Backtrace:
>   "re-search-forward" (0x8898d0)
>   "c-syntactic-re-search-forward" (0x889ed0)
>   "c-forward-declarator" (0x88a410)
>   "c-font-lock-declarators" (0x88a980)
>   "c-font-lock-single-decl" (0x88ae50)
>   0xad881a0 PVEC_COMPILED
>   "c-find-decl-spots" (0x88c040)
>   "c-font-lock-declarations" (0x88c410)
>   "font-lock-fontify-keywords-region" (0x88ca70)
>   "font-lock-default-fontify-region" (0x88ce60)
>   "c-font-lock-fontify-region" (0x88d230)
>   "font-lock-fontify-region" (0x88d578)
>   0x83d89a8 PVEC_COMPILED
>   "run-hook-wrapped" (0x88daf0)
>   "jit-lock--run-functions" (0x88dee0)
>   "jit-lock-fontify-now" (0x88e3e0)
>   "jit-lock-stealth-fontify" (0x88e9d0)
>   "apply" (0x88e9c8)
>   "timer-event-handler" (0x88edb8)
>   (gdb) p n
>   $1 = 1
>   (gdb) p lim
>   $2 = <optimized out>
>   (gdb) pp bound
>   123806
>   (gdb) p PT
>   $3 = 123811

> So point is 123811 and the BOUND argument of re-search-forward is
> 123806, too small.

What I think's happening is that c-forward-declarator has found a "["
which is before BOUND, but then sets point to the matching "]" which is
after BOUND.  It then calls c-syntactic-re-search-forward again,
resulting in the error.

In master's process.c, there is a "]" very close to 123811.

>   (gdb) up
>   #1  0x011cd2c9 in Fre_search_forward (regexp=XIL(0x800000000ad97598),
>       bound=..., noerror=..., count=...) at search.c:2271
>   2271      return search_command (regexp, bound, noerror, count, 1, 1, 0);
>   (gdb) pp regexp
>   "[;:,]\\|\\s)\\|\\(=\\|\\s(\\)"
>   (gdb) p current_buffer
>   $4 = (struct buffer *) 0xb362590
>   (gdb) pp current_buffer->name_
>   "process.c"

> These are the regexp argument to re-search-forward and the buffer
> which was being fontified.

> The problem happens in c-syntactic-re-search-forward in this snippet:

[ .... ]

> This is called from c-forward-declarator:

>        ;; Search syntactically to the end of the declarator (";",
>        ;; ",", a closing paren, eob etc) or to the beginning of an
>        ;; initializer or function prototype ("=" or "\\s\(").
>        ;; Note that square brackets are now not also treated as
>        ;; initializers, since this broke when there were also
>        ;; initializing brace lists.
>        (let (found)
>          (while
>              (and (progn
>                     ;; In the next loop, we keep searching forward whilst
>                     ;; we find ":"s which aren't single colons inside C++
>                     ;; "for" statements.
>                     (while
>                         (and
>                          (setq found
>                                (c-syntactic-re-search-forward   <<<<<<<<<<<
>                                 "[;:,]\\|\\s)\\|\\(=\\|\\s(\\)"
>                                 limit t t))

As already suggested, I think the bug is that there aren't enough checks
that (< (point) limit) in this function.  I have added them in.

> It looks like c-syntactic-re-search-forward calls re-search-forward in
> a loop, but perhaps it fails to update the limit to be in sync with
> point that moves as the search proceeds?

A further problem is that c-font-lock-declarators is calling
c-forward-declarator with a limit; this is silly - if the end of a
declaration runs over a jit-lock chunk boundary, we still want to
fontify this declaration fully.  So I've changed the LIMIT argument in
the pertinent two calls to nil.  (There is a third call somewhere where
this LIMIT argument is the end of a macro, and it is absolutely needed).

> Let me know what other data I can provide to help fix this annoying
> problem.

I haven't reproduced the problem, but I admit I haven't tried all that
hard.  Could you please try out the patch below, and let me know if it
fixes the bug.

> In GNU Emacs 26.0.90 (build 1, i686-pc-mingw32)
>  of 2017-10-12 built on HOME-C4E4A596F7
> Windowing system distributor 'Microsoft Corp.', version 5.1.2600
> Recent messages:
> For information about GNU Emacs and the GNU system, type C-h C-a.

[ .... ]


diff --git a/lisp/progmodes/cc-engine.el b/lisp/progmodes/cc-engine.el
index 3792835752..07b9215046 100644
--- a/lisp/progmodes/cc-engine.el
+++ b/lisp/progmodes/cc-engine.el
@@ -8102,12 +8102,14 @@ c-forward-declarator
         ;; initializing brace lists.
         (let (found)
           (while
-              (and (progn
+              (and (< (point) limit)
+                   (progn
                      ;; In the next loop, we keep searching forward whilst
                      ;; we find ":"s which aren't single colons inside C++
                      ;; "for" statements.
                      (while
                          (and
+                          (< (point) limit)
                           (setq found
                                 (c-syntactic-re-search-forward
                                  "[;:,]\\|\\s)\\|\\(=\\|\\s(\\)"
@@ -8129,7 +8131,7 @@ c-forward-declarator
                    (c-go-up-list-forward))
             (setq brackets-after-id t))
           (when found (backward-char))
-          t))
+          (<= (point) limit)))
        (list id-start id-end brackets-after-id (match-beginning 1) decorated)
 
       (goto-char here)
diff --git a/lisp/progmodes/cc-fonts.el b/lisp/progmodes/cc-fonts.el
index 02b685d240..b8dbe3c26b 100644
--- a/lisp/progmodes/cc-fonts.el
+++ b/lisp/progmodes/cc-fonts.el
@@ -1062,7 +1062,7 @@ c-font-lock-declarators
     ;; The following `while' fontifies a single declarator id each time round.
     ;; It loops only when LIST is non-nil.
     (while
-       (and pos (setq decl-res (c-forward-declarator limit)))
+       (and pos (setq decl-res (c-forward-declarator)))
       (setq next-pos (point)
            id-start (car decl-res)
            id-face (if (and (eq (char-after) ?\()
@@ -1091,7 +1091,7 @@ c-font-lock-declarators
                                               (throw 'is-function nil))
                                              ((not (eq got-type 'maybe))
                                               (throw 'is-function t)))
-                                            (c-forward-declarator limit t)
+                                            (c-forward-declarator nil t)
                                             (eq (char-after) ?,))
                                         (forward-char)
                                         (c-forward-syntactic-ws))


-- 
Alan Mackenzie (Nuremberg, Germany).