bug#28850: 26.0.90; Error running timer 'jit-lock-stealth-fontify': (error "Invalid search bound (wrong side of point)")

[Alan Mackenzie]

Hello, Eli.

On Tue, Oct 24, 2017 at 17:46:08 +0300, Eli Zaretskii wrote:
> > Date: Sun, 22 Oct 2017 20:13:40 +0000
> > Cc: 28850@debbugs.gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > So point is 123811 and the BOUND argument of re-search-forward is
> > > 123806, too small.

> > What I think's happening is that c-forward-declarator has found a "["
> > which is before BOUND, but then sets point to the matching "]" which is
> > after BOUND.  It then calls c-syntactic-re-search-forward again,
> > resulting in the error.

> > In master's process.c, there is a "]" very close to 123811.

> For the record, here's the place where this happened, with the two
> locations shown by "^":

>       char namebuf[sizeof (ifq->ifr_name) + 1];
>                                          ^    ^
> The reason you don't see this in process.c you have is that the
> version I used was edited wrt to what you have.

:-).  It's exactly as I'd surmised.

> > I haven't reproduced the problem, but I admit I haven't tried all that
> > hard.  Could you please try out the patch below, and let me know if it
> > fixes the bug.

> Thanks, this fixes a very large part of the problem, so I think you
> should install this on the release branch.

I'll do that just as soon as I'm in the mood for routine work.  This
evening was right for debugging the rest of it.

> It doesn't solve all of it, though, because I got that breakpoint hit
> again.  This time it took much longer before that happened (a sign
> that most of the problem is indeed solved), and the backtrace is
> different.

This is an entirely separate bug.

> Here's the C and the Lisp backtraces, followed by some relevant
> values:

[ .... ].

>   Lisp Backtrace:
>   "re-search-forward" (0x88c050)
>   0xad7f328 PVEC_COMPILED
>   "font-lock-fontify-keywords-region" (0x88ca70)
>   "font-lock-default-fontify-region" (0x88ce60)
>   "c-font-lock-fontify-region" (0x88d230)
>   "font-lock-fontify-region" (0x88d578)
>   0x13810840 PVEC_COMPILED
>   "run-hook-wrapped" (0x88daf0)
>   "jit-lock--run-functions" (0x88dee0)
>   "jit-lock-fontify-now" (0x88e3e0)
>   "jit-lock-stealth-fontify" (0x88e9d0)
>   "apply" (0x88e9c8)
>   "timer-event-handler" (0x88edb8)
>   (gdb) pp current_buffer->name_
>   "platform.h"
>   (gdb) pp current_buffer->directory_
>   "d:/utils/lz4-1.7.5/programs/"
>   (gdb) p PT
>   $1 = 2645
>   (gdb) p bound
>   $2 = make_number(2425)
>   (gdb) up
>   #1  0x011cd2c9 in Fre_search_forward (regexp=XIL(0x800000000adcb1f0),
>       bound=..., noerror=..., count=...) at search.c:2271
>   2271      return search_command (regexp, bound, noerror, count, 1, 1, 0);
>   (gdb) pp regexp
>   "\\(\\=\\|\\(\\=\\|[^\\]\\)[
>   ]\\)\\s *#\\s 
> *\\(\\(?:\\(?:el\\)?if\\)\\)\\([^[:alnum:]_$]\\|$\\)\\(\\\\\\(.\\|
>   [
>   ]\\)\\|[^
>   ]\\)*"

I've tracked down that regexp.  It matches "#if" or "#elif" inside
macros.  It's used in c-cpp-matchers in cc-fonts.el.

> As you see, point is at 2645, whereas BOUND is at 2425.  This happens
> in the file platform.h from the lz4-1.7.5 distribution.  Here's the
> relevant part of platform.h with the two locations shown (I added an
> empty line for each "^" marker):

The reason for this is that a generated lambda form with the argument
LIMIT (which would be the end of a jit-lock chunk or similar) internally
binds LIMIT to the end of the current macro.  Inside this binding, which
searches for "defined" repeatedly, we go forward to after the last
"defined", as indicated in your source excerpt below.  Unfortunately,
this point is beyond the original LIMIT supplied to the lambda, so in the
next re-search-forward, point is the wrong side of this original LIMIT.

This particular bit of CC Mode is "write only" code, and it could take
me some while to disentangle the stack of macros and function generators
which has produced it.  I think I'm going to extract this form and
rewrite it more by hand, making it simpler to debug in the future.

> /* **************************************
> *  Detect 64-bit OS
> *  
> http://nadeausoftware.com/articles/2012/02/c_c_tip_how_detect_processor_type_using_compiler_predefined_macros
> ****************************************/
> #if defined __ia64 || defined _M_IA64                                         
>                                       /* Intel Itanium */ \
>   || defined __powerpc64__ || defined __ppc64__ || defined __PPC64__          
>                                       /* POWER 64-bit */  \
>   || (defined __sparc && (defined __sparcv9 || defined __sparc_v9__ || 
> defined __arch64__)) || defined __sparc64__  /* SPARC 64-bit */  \
>   || defined __x86_64__s || defined _M_X64                                    
>                                       /* x86 64-bit */    \
>   || defined __arm64__ || defined __aarch64__ || defined __ARM64_ARCH_8__     
>                                       /* ARM 64-bit */    \
>   || (defined __mips  && (__mips == 64 || __mips == 4 || __mips == 3))        
>                                       /* MIPS 64-bit */   \
>                                                                       ^
>   || defined _LP64 || defined __LP64__ /* NetBSD, OpenBSD */ || defined 
> __64BIT__ /* AIX */ || defined _ADDR64 /* Cray */               \
>   || (defined __SIZEOF_POINTER__ && __SIZEOF_POINTER__ == 8) /* gcc */
>               ^
> #  if !defined(__64BIT__)
> #    define __64BIT__  1
> #  endif
> #endif

> As you see, BOUND is after the closing paren, after "3))", and point
> is at the beginning of "__SIZEOF_POINTER__" a couple of lines further.

Yes.  BOUND would have been the end of a jit-lock-chunk.

> To tell the truth, the Lisp backtrace puzzles me a bit, because the
> only call to re-search-forward in font-lock-fontify-keywords-region is
> protected by a condition that should have prevented this problem from
> happening:

>       (while (and (< (point) end)
>                 (if (stringp matcher)
>                     (re-search-forward matcher end t)
>                   (funcall matcher end))

> So maybe I'm missing something, or maybe the problematic call to
> re-search-forward comes from some macro expansion I didn't identify.

Indeed it does.  The expansion of the macro is in C Mode's
font-lock-keywords.

> Let me know if I can provide any more details for your analysis.

Will do, but I think you've given me enough to solve this.  Thanks!

> Thanks.

-- 
Alan Mackenzie (Nuremberg, Germany).