confusion over undocumented syntax-table features, font-lock and syntax-tables

[Luc Teirlinck]

Matthew Smith wrote:

    The results of the following code completely baffles me.  Is
    global-font-lock-mode changing the syntax classes?

Yes, using syntax text properties, and it is impossible to get the
correct shell syntax without such text properties.  (However does
anybody understand the behavior of char-syntax in the ielm run
below???)

Did you specify which shell you are using?  If it is bash, then #
starts a comment at the beginning of a word, elsewhere it has symbol
syntax.

    I was observing a strange behavior in `sh-mode' defined in
    sh-script.el where (re-search-forward "\\s<\\s<") was failing even
    though it was passing over a buffer substring of two characters
    whose syntax classes, as reported by `(char-syntax (char-after
    N))' and N+1 was "<".

I do not know how you possibly can get two consecutive characters with
comment-start syntax in bash. (I do not know about other shells.)

I cut out the "test file" you included (see below) and put point at
the beginning of the line:

## boln is at buffer position 40

Then I ran ielm (for convenience, if you prefer, ypu can use M-:).

The result shows the (correcting) influence of font-lock-mode:
(Ran using:
 emacs-21.3.50 -q --no-site-file --eval '(blink-cursor-mode 0)' &
This is today's CVS.)

Remember that, in the syntax-after return values, 3 stands for symbol,
11 for comment-start.

===File ~/shellsyntax=======================================
*** Welcome to IELM ***  Type (describe-mode) for help.
ELISP> (set-buffer "testfile.sh")  ;; ielm specific code
#<buffer testfile.sh>
ELISP> (current-buffer)
#<buffer testfile.sh>
ELISP> (point)
40
ELISP> parse-sexp-ignore-comments
t
ELISP> (string (char-syntax (point)))
"("  ;; goes completely over my head
ELISP> (string (char-syntax (1+ (point))))
")"  ;; this too
ELISP> (get-char-property (point) 'syntax-table)
nil
ELISP> (get-char-property (1+ (point)) 'syntax-table)
nil
ELISP> (syntax-after (point))
(11)  ;; I understand this.

ELISP> (syntax-after (1+ (point)))
(11)  ;; This too, even though it is wrong.

ELISP> (global-font-lock-mode 1)
t
ELISP> (string (char-syntax (point)))
"("  ;; ???
ELISP> (string (char-syntax (1+ (point))))
")"  ;; ???
ELISP> (get-char-property (point) 'syntax-table)
nil
ELISP> (get-char-property (1+ (point)) 'syntax-table)
(3)  ;; font-lock-mode to the rescue

ELISP> (syntax-after (point))
(11)  

ELISP> (syntax-after (1+ (point)))
(3)  ;; correct: the second # has symbol syntax,
     ;; it does not start a comment.

ELISP> sh-shell
bash
ELISP> sh-shell-file
"/usr/local/bin/bash"
ELISP> (string (char-syntax ?#))
"<"  ;; I undestand this too, but how does this rhyme with the above???
ELISP> ============================================================

Do not ask me to explain the char-syntax behavior.  I have no clue.

Sincerely,

Luc.

Appendix:

Test file used:

-----cut here
(setq test "
hello () { echo world.; }
## boln is at buffer position 40
")
(defun test ()
  (sh-mode)
  (message "result is %S"
           (if (and
                (equal "<" (char-to-string (char-syntax ?#)))
                (equal (char-after 40) ?#)
                (equal (char-after 41) ?#)
                (equal "<" (char-to-string (char-syntax (char-after 40))))
                (equal "<" (char-to-string (char-syntax (char-after 41))))
                )
           (save-excursion
             (goto-char (point-min))
             (re-search-forward "\\s<\\s<"))
         "whoops!")))
(progn
  (global-font-lock-mode 0)
  ;; succeeds
  (test))
(progn
  (global-font-lock-mode 1)
  ;; `re-search-forward' fails the SECOND time, if not the first (no
  ;; pattern found)
  (test))

;;(sh-mode)
;;(emacs-lisp-mode)
;;(global-font-lock-mode)
;;(test)
---- end of test file