RE: paren match highlighting is slightly broke

[Bielawski, Richard G.]

Exterminators,

Here is some follow-up info that might be useful in finding the problem.
#2 mentions a possible separate bug easily recreated with this code.

1) When the problem occurs forward-sexp still operates correctly.  
   That is it does find the proper end statement.

2) backward-sexp operates correctly only if the comment does not contain
syntactic parens.
   If it does, they are not ignored like they are in forward-sexp.
   This appears to be a separate bug creatable with the following line in a
tal-mode buffer.
   BEGIN   BEGIN   BEGIN  !END!  END  END   END

3) The following is the actual code I used to set the syntax-table
properties in TAL-MODE.
   Perhaps useful in recreating the problems rather than contriving your
own.
   The entire tal-mode.el is available from emacs-wiki.

(defvar tal-font-lock-syntactic-keywords
 '(;; The B of Begin gets open paren syntax.  Case-fold is assumed true.
   ("\\(^\\|\\s \\)\\(b\\)egin\\(\\s-\\|$\\)" (2 (4 . ?d)))
   ;; The D of End gets close paren syntax.
   ("\\(\\s-\\|^\\)en\\(d\\)\\b" (2 (5 . ?B)))
   ;; The function returns matches 1&2 for comments, 3&4 for strings
   (tal-find-syntactic-keywords
    (1 (11) nil t) (2 (12) nil t) (3 (15) nil t) (4 (15) nil t))
  )
 "A list of regexp's or functions.  Used to add syntax-table properties to
characters that can't be set by the syntax-table alone."
)
(defun tal-find-syntactic-keywords ( search-limit )
  ;; Comments starting with -- go to eol always while comments starting with
! 
  ;; go until another ! or eol.  Strings start and end with " as usual but
can't
  ;; span lines so they are terminated by eol.  This fcn returns nil if
neither 
  ;; is found.  True is returned and match-string 1&2 are set if a comment
is found.
  ;; True is returned and match-string 3&4 are set if a string is found.  
  ;; Point is left at the end of the match.
  (when (re-search-forward "\\(?:--\\|!\\|\"\\)" search-limit t)
    (let ((start (match-data))
          (match (match-string-no-properties 0))
          end)
      (if (equal "\"" match)    
          ;string found.  Find ending quote or use eol as end
          (when (re-search-forward "\\(?:\n\\|\"\\|\\'\\)" search-limit t)
            (setq end (match-data))                   
            (set-match-data
             `(,(car start) ,(car (cdr end))    ;match-string 0
               nil nil nil nil                  ;match-string 1&2 not found
               ,@start ,@end)))                 ;match-string 3&4
        ;; Must be a comment.  Determine type.
        (if (equal "!" match)
            (setq match "\\(?:\n\\|!\\|\\'\\)") ;! can delimit as well as
eol/eob
          (setq match "\\(?:\n\\|\\'\\)"))      ;only eol or eob delimits
        ;;see above 'when' comment
        (when (re-search-forward match search-limit t) 
          (setq end (match-data))
          (set-match-data
           `(,(car start) ,(car (cdr end))      ;match-string 0
             ,@start ,@end)))))                 ;match-string 1&2
    t)
)
-----Original Message-----
From: Bielawski, Richard G. 
Sent: Thursday, August 12, 2004 7:42 PM
To: 'Emacs-Pretest-Bug (E-mail)
Subject: paren match highlighting is slightly broke


In GNU Emacs 21.3.50.1 (i386-mingw-nt5.0.2195) of 2004-07-26 on BERATUNG4
configured using `configure --with-gcc (3.3) --cflags
-I../../jpeg-6b-1/include -I../../libpng-1.2.4-1/include
-I../../tiff-3.5.7/include -I../../xpm-nox-4.2.0/include
-I../../zlib-1.1.4-1/include'

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: ENU
  locale-coding-system: cp1252
  default-enable-multibyte-characters: t

Major mode: Lisp Interaction

Minor modes in effect:
  eldoc-mode: t
  cua-mode: t
  recentf-mode: t
  show-paren-mode: t
  which-function-mode: t
  tool-bar-mode: t
  encoded-kbd-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  unify-8859-on-encoding-mode: t
  utf-translate-cjk-mode: t
  line-number-mode: t
  transient-mark-mode: t

Paren match highlighting often does not work when syntax is set with 
font-lock-syntactic-keywords.
The following 2 lines comprise the entire test file

-- turn -*-tal-*- mode on here.
BEGIN   BEGIN   BEGIN   END  !TEST!  END   END

I've included the properties of these 2 lines below.
In prior versions (21.2.1 for example) this worked fine.  
In this version, placing the cursor before any of the BEGIN keywords
causes the corresponding END to light up ONLY if there are no intervening 
comments.  In the example, only the matching pair prior to the !TEST!
comment light up.  Not even the B lights up as unmatched on the other
2 pairs.  It doesn't seem to make any difference where I put the comment.
Comments effectively prevent paren matching from occurring.

It may help to know that the problem also occurs in this example

BEGIN   BEGIN () !TEST!   BEGIN   END  END   END

but in this next example

BEGIN   BEGIN (!test!) !TEST!   BEGIN   END  END   END

Everything works fine.  All the begin/end pairs light up correctly.

Here are the properties of the original 2 lines.

#("-- turn -*-tal-*- mode on here.
"
0 2 (face font-lock-comment-face syntax-table (11) fontified t)
2 31 (face font-lock-comment-face fontified t)
31 32 (face font-lock-comment-face syntax-table (12) fontified t))
#("begin  begin  begin  end ! end ! end
"
0 1 (face font-lock-keyword-face syntax-table (4 . 100) fontified t)
1 5 (face font-lock-keyword-face fontified t)
5 7 (fontified t)
7 8 (face font-lock-keyword-face syntax-table (4 . 100) fontified t)
8 12 (face font-lock-keyword-face fontified t)
12 14 (fontified t)
14 15 (face font-lock-keyword-face syntax-table (4 . 100) fontified t)
15 19 (face font-lock-keyword-face fontified t)
19 21 (fontified t)
21 23 (face font-lock-keyword-face fontified t)
23 24 (face font-lock-keyword-face syntax-table (5 . 66) fontified t)
24 25 (fontified t)
25 26 (face font-lock-comment-face syntax-table (11) fontified t)
26 29 (face font-lock-comment-face fontified t)
29 30 (face font-lock-comment-face syntax-table (5 . 66) fontified t)
30 31 (face font-lock-comment-face fontified t)
31 32 (face font-lock-comment-face syntax-table (12) fontified t)
32 33 (fontified t)
33 35 (face font-lock-keyword-face fontified t)
35 36 (face font-lock-keyword-face syntax-table (5 . 66) fontified t)
36 37 (fontified t))

Richard Bielawski
612-667-5039