RE: Clarification needed: syntax-propertize vs font-lock-syntax-table

[Wedler, Christoph]

>> And syntax-propertize should be called at more places.  Currently, doing
>> it lazily does not really work, as it is not called when needed.

> You'll have to be more specific.

Well, I realized that (with Emacs-24.4) the imenu list in antlr-mode
wasn't correctly before I put (syntax-propertize (point-max)) into
antlr-imenu-create-index-function.  I could imagine similar things in
other modes.

I have not seen other issues, but I do use font-lock (which also calls
syntax-propertize), as it is the default.  I could imagine that issues
could appear for people who have switched off font-lock.  (Needs to be
tested.)

>> That is why js-mode calls (syntax-propertize (point-max)) at the end
>>

> Y Not anymore, it doesn't.

You are right...

>> (and I will do the same in antlr-mode).

> I'd prefer if you didn't.

I probably define a variable for that which will be t for Emacs below
emacs-25.x and nil with the next release.

>> My other question was concerning (nth 9 ppss), i.e. the list of open
>> parentheses is very useful to have (and its also used in syntax.el
>> itself) -> I would like to have this to be official in the lisp
>> docstring as well - in src/syntax.c, we have

> I thought that's exactly what you meant by documenting the 9th element. 
> Were there any other options?

The other documentation request was for font-lock syntax-table as being
"syntax-ppss-compatible" (as you put it nicely).

> I'm not the guy who can say if it's okay or not. Probably is. But it 
> would be sad if after that we choose a way to adapt syntax-ppss for the 
> multiple-modes case, and it will require saving a more complex structure 
> in the 9th element.

Agreed.  How about a function like

(defun syntax-ppss-open-list-positions (ppss)
  "Get all syntactically open list position found in a syntactic scan.
PPSS is a scan state, as returned by `parse-partial-sexp' or `syntax-ppss'.
The start position of the outermost list comes first."
  (nth 9 ppss))

This function needs to be changed if the internal structure of (nth 9
ppss) is changed.

>> In the case of antlr-mode (grammar with actions), it could be done as
>> long as it does not break the calulation of the "inner-mode-chunk-end",
>> e.g, with Python actions
>>
>>     rule: "KEYWORD" { do_something {  } # do it };
>>
>> the action ends with the final "}" (even though a naive Python-like
>> syntax-propertization would think of it as part of a comment)

> Yes, of course we should try to make it break as few things as possible. 
> Not sure if I could comment on this in any more detail.

We might need something (an extra syntax flag?) which makes it easy to
"rewind" the ppss such that

  (equal (syntax-ppss inner-mode-end-position)
         (syntax-ppss inner-mode-start-position))