Problems with syntax-ppss: Was [... Apply `comment-depth' text properties when calling `back_comment'.]

[Alan Mackenzie]

Hello again, Stefan.

On Tue, Mar 08, 2016 at 07:17:17PM -0500, Stefan Monnier wrote:
> > syntax-ppss is a lisp level thing, rather loosely defined.

> No, it's pretty precisely defined: it (should) returns the same value as
> (parse-partial-sexp (point-min) pos).

I note the lack of confidence expressed by the word "should".  Also we
should surely want what the result of a scan starting at position 1 (not
(point-min)) would be.

> > For example it reacts to the setting/clearing of the particular
> > syntax-table text properties it's sensitive to, even when
> > inhibit-modification-hooks is non-nil (which it _always_ is when text
> > properties are changed).

> `syntax-ppss' is used all over the place, so if there's a problem with it,
> we should fix it.

There are several problems with syntax-ppss.  It is not a rigorously
correct function (which is what we would expect from this low level of
the Emacs functionality.)  Looking at the version in the master branch:

Minor problems:
(i) After inserting a `with-syntax-table' near the beginning of
  syntax-ppss, the indentation wasn't redone.
(ii) The cache isn't flushed when a syntax-table text property gets set
  or cleared (see my patch for how to do this).
(iii) The specification of the function states that the returned state
  is equivalent to a scan from (point-min) rather than 1.  This parse
  state isn't very useful, and probably forces callers to widen before
  calling, or at least ensure that (point-min) isn't in a literal.

Major problems:
(i) The cache doesn't appear to be flushed on narrowing or widening the
  buffer.
(ii) No account is taken of possibly starting a parse-partial-sexp from
  after the first character of a two character comment starter or ender.
  OK, this is really a problem with the parser state not recording this
  situation (which we could correct), but failing that, syntax-ppss
  should check for this.  We did talk about this (in connection with a
  CC Mode bug) a few years ago.

Also the function is complicated and difficult to understand.  It seems
to be fairly heavily optimised.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).