bug#15045: Point jumps inappropriately around time of Semantic lexing

[Barry OReilly]

See this thread for the original description:

   http://lists.gnu.org/archive/html/emacs-devel/2013-07/msg00328.html

The short of it is that I occasionally witness point jump to elsewhere
in the buffer while I'm editing and then back again. The change of
point is visually apparent in the display. This has consequences such
as causing undesired scrolling while editing.

To debug it, I used a change like this:

diff --git a/src/editfns.c b/src/editfns.c
index 50bde90..039e13f 100644
--- a/src/editfns.c
+++ b/src/editfns.c
@@ -234,6 +234,8 @@ The return value is POSITION.  */)
   (register Lisp_Object position)
 {
   ptrdiff_t pos;
+  bool noninteractive_old = noninteractive;
+  noninteractive = true;

   if (MARKERP (position)
       && current_buffer == XMARKER (position)->buffer)
@@ -245,7 +247,19 @@ The return value is POSITION.  */)
        SET_PT_BOTH (ZV, ZV_BYTE);
       else
        SET_PT_BOTH (pos, marker_byte_position (position));
-
+      int btI = 0;
+      Lisp_Object theBacktrace;
+      do
+      {
+         theBacktrace = Fbacktrace_frame( make_number(btI) );
+         ++btI;
+         Fprin1(theBacktrace, Qnil);
+         printf("\017\n");
+      } while( ! EQ(theBacktrace, Qnil) );
+      { struct timespec debug_ts; char debug_dateStr[20]; {
clock_gettime(CLOCK_REALTIME, &debug_ts); struct tm mytm;
localtime_r(&debug_ts.tv_sec, &mytm); strftime(debug_dateStr, 20,
"%Y-%m-%dT%H:%M:%S", &mytm); }
+           printf( "%s.%09ld|pid:%d|tid:%ld|%s|%d| DEBUG: goto-char
marker pos=%ld\n", // TODO: debugging
+                             debug_dateStr, debug_ts.tv_nsec,
getpid(), pthread_self(), __FILE__, __LINE__, pos ); fflush(stdout); }
+      noninteractive = noninteractive_old;
       return position;
     }

@@ -253,6 +267,19 @@ The return value is POSITION.  */)

   pos = clip_to_bounds (BEGV, XINT (position), ZV);
   SET_PT (pos);
+   int btI = 0;
+   Lisp_Object theBacktrace;
+   do
+   {
+      theBacktrace = Fbacktrace_frame( make_number(btI) );
+      ++btI;
+      Fprin1(theBacktrace, Qnil);
+      printf("\017\n");
+   } while( ! EQ(theBacktrace, Qnil) );
+   { struct timespec debug_ts; char debug_dateStr[20]; {
clock_gettime(CLOCK_REALTIME, &debug_ts); struct tm mytm;
localtime_r(&debug_ts.tv_sec, &mytm); strftime(debug_dateStr, 20,
"%Y-%m-%dT%H:%M:%S", &mytm); }
+        printf( "%s.%09ld|pid:%d|tid:%ld|%s|%d| DEBUG: goto-char
position pos=%ld\n", // TODO: debugging
+                          debug_dateStr, debug_ts.tv_nsec, getpid(),
pthread_self(), __FILE__, __LINE__, pos ); fflush(stdout); }
+   noninteractive = noninteractive_old;
   return position;
 }


I got a reproduction with these debug statements. I was editing a
Makefile in comments at about buffer position 3396 and observed point
temporarily jump to the beginning of the comment block at position
3084. I filtered the output for calls to go to position 3084 around
the time I witnessed this:

   [backtrace A]
   2013-08-01T10:37:28.119778000|pid:10485|tid:2342111488|editfns.c|281|
DEBUG: goto-char position pos=3084
   [backtrace B]
   2013-08-01T10:37:28.412962000|pid:10485|tid:2342111488|editfns.c|261|
DEBUG: goto-char marker pos=3084
   [backtrace C]
   2013-08-01T10:37:29.715413000|pid:10485|tid:2342111488|editfns.c|281|
DEBUG: goto-char position pos=3084

Strangly, the backtrace A and B are 35 and 45 empty (except my Ctrl-O
chars) stack frames respectively. Backtrace C is:

(t semantic-make-lexer 3083 3257 nil nil)^O
(t semantic-lex 3083 3257 nil)^O
(t semantic-parse-region-default 3083 3257 nil nil nil)^O
(t semantic-parse-region 3083 3257 nil)^O
(t semantic-edits-incremental-parser-1)^O
parser error: %S" error-message-string t] 4)))] 3)^O
(t semantic-parse-changes-default)^O
(t semantic-parse-changes)^O
(t semantic-fetch-tags)^O
(t byte-code "<U+008A><U+008A>À <U+0088>*Á<U+0087>"
[semantic-fetch-tags nil] 1)^O
(t semantic-idle-scheduler-refresh-tags)^O
(t byte-code "Æ^XÇpÇÆÈÉÊ
\"\"\"^YÆ<U+0089>^ZESCÆ^\^M;<U+0085>^\^@Ë^M!^^#^N$<U+0085>^^@ÌÍ!<U+0085>+^@^N^M?<U+0085>^^@^N%?<U+0085>^^@^N#<U+0084>E^@^M;<U+0083>E^@Î^M!<U+0084>R^@^N#<U+0085>^^@Ë^MÆÏ#<U+0085>^^@^N&ÐX<U+0086>^
^M;<U+0085>­^@Ë^M!^^#^N$<U+0085>ï^@ÌÍ!<U+0085>¼^@^N^M?<U+0085>ï^@^N%?<U+0085>ï^@^N#<U+0084>Ö^@^M;<U+0083>Ö^@Î^M!<U+0084>ã^@^N#<U+0085>ï^@Ë^MÆÏ#<U+0085>ï^@^N&ÐX<U+0086>ï^@Ñ
^N&W)<U+0083>
^A^N(<U+0083>þ^@Ò <U+0088><U+0082>
^AÓÚÛ<U+008F><U+0088><U+0082>
^Ap^KB^S)^N*A<U+0089>^V*<U+0084>^?^@*^K^Q)^N,Æ^^-<U+0089>^^*<U+0083>o^A^N*@^V-<U+008A>^N+<U+0083>>^AÖ
<U+0084>8^A× 
<U+0083>>^AØ^N+Ü\"<U+0088>^N.<U+0083>I^AÝÞ^N-\"<U+0088>^N(<U+0083>U^A^N-
<U+0088><U+0082>Z^AÓßà<U+008F>
e-scheduler-queue service semantic-idle-scheduler-verbose-flag] 8)^O
(t semantic-idle-core-handler)^O
(t semantic-idle-scheduler-function)^O
(t apply semantic-idle-scheduler-function nil)^O
(t byte-code "rÂÃH\"<U+0088>)Á<U+0087>" [timer apply 5 6] 4)^O
(t timer-event-handler [t 0 1 0 t semantic-idle-scheduler-function nil
idle 0])^O
nil^O

I can see the call to goto-char in define-lex (the macro which creates
semantic-make-lexer). However, there is a save-excursion in effect at
the semantic-idle-core-handler frame, so this goto-char wouldn't seem
to be the same goto-char I observe in the display.

I'm not sure about the empty backtraces. Is the code I used to print
backtraces valid? In my many runs, empty backtraces are very rare.

I have since started using Fbacktrace() instead of the more long
winded code above. Unfortunately I'm having a reproduction drought in
the past week.

One additional observation not previously noted is that I see this bug
much more often when editing comments or strings. However, I'm fairly
sure I've seen it when editing straight code too.

In GNU Emacs 24.3.50.1 (x86_64-unknown-linux-gnu, GTK+ Version 2.10.4)
 of 2013-06-18 on psd15
Windowing system distributor `The X.Org Foundation', version 11.0.70101000
System Description:     Red Hat Enterprise Linux Client release 5.4 (Tikanga)

Configured using:
 `configure
 
--prefix=/redacted/user/boreilly/sw/emacs-install-trunk-20899d085afe62520113b5acbfe3dbba57823dc9
 --with-gif=no'

Important settings:
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=none
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

Major mode: Lisp Interaction

Minor modes in effect:
  outline-minor-mode: t
  global-whitespace-mode: t
  global-ede-mode: t
  global-semanticdb-minor-mode: t
  global-semantic-idle-scheduler-mode: t
  semantic-mode: t
  evil-mode: t
  evil-local-mode: t
  global-undo-tree-mode: t
  undo-tree-mode: t
  show-paren-mode: t
  delete-selection-mode: t
  global-auto-revert-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
M-x r e p o r t - <tab> <return>

Recent messages:
Loading redacted
Loading redacted
Loading whitespace...done
2013-08-07T13:47:52.565183 Loading tags file: redacted
Starting a new list of tags tables
2013-08-07T13:47:52.716876 Finished loading init file.
2013-08-07T13:47:52.723671 Inside my-prog-mode-hook
2013-08-07T13:47:52.732424 Inside my-emacs-lisp-mode-hook for buffer *scratch*
For information about GNU Emacs and the GNU system, type C-h C-a.
2013-08-07T13:47:52.748012 ---------------- Finished with
my-emacs-startup-hook. ----------------

Load-path shadows:
None found.

Features:
(shadow sort gnus-util mail-extr emacsbug message format-spec rfc822 mml
mml-sec mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev
gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums mm-util
mail-prsvr mail-utils noutline outline easy-mmode my-config warnings
semantic/lex-spp etags package cl-macs whitespace cus-start cus-load
my-proj ede/cpp-root ede/speedbar ede/files ede ede/base ede/auto
ede/source eieio-speedbar speedbar sb-image dframe eieio-custom wid-edit
semantic/db-mode semantic/idle semantic/bovine/gcc semantic/dep
semantic/ia semantic/analyze/refs semantic/db-find semantic/db-ref
semantic/senator semantic/decorate pulse semantic/analyze semantic/sort
semantic/scope semantic/analyze/fcn semantic/db gv eieio-base
semantic/ctxt semantic/format ezimage semantic/tag-ls semantic/find
semantic/util-modes easymenu semantic/util semantic semantic/tag
semantic/lex semantic/fw eieio byte-opt bytecomp byte-compile cconv
eieio-core mode-local cedet evil evil-integration evil-maps
evil-commands evil-types evil-search evil-ex evil-macros evil-repeat
evil-states evil-core evil-common windmove rect evil-digraphs evil-vars
ring edmacro kmacro undo-tree diff goto-chg rainbow-delimiters my-util
advice help-fns electric paren delsel autorevert cl nadvice cl-lib
time-date tooltip ediff-hook vc-hooks lisp-float-type mwheel x-win x-dnd
tool-bar dnd fontset image regexp-opt fringe tabulated-list newcomment
lisp-mode register page menu-bar rfn-eshadow timer select scroll-bar
mouse jit-lock font-lock syntax facemenu font-core frame cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese case-table epa-hook jka-cmpr-hook help simple abbrev
minibuffer loaddefs button faces cus-face macroexp files text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote make-network-process dbusbind
dynamic-setting system-font-setting font-render-setting move-toolbar gtk
x-toolkit x multi-tty emacs)