Re: Bug #25608 and the comment-cache branch

[Stefan Monnier]

> I see, thanks. And I think that means that, ideally, it would work without
> the caller having to adjust the syntax visibility bounds, or the like, as
> long as the syntax table is correct and the beginning (or the end) of the
> currently navigated comment is within view.

Right, but not reliably so: very often we need to parse backward not
just until the matching starter but until the previous closer (to make
sure the starter we saw was not itself within an earlier comment), and
in other cases the mix of comment markers and string markers make it
impossible to guess if we were really inside a comment, so we end up
falling back on the forward-parse code.

>> In the case we do scan forward (e.g. the case where we end up using
>> parse-partial-sexp (or syntax-ppss in my patch)), we actually manually
>> re-introduce that behavior: if the forward parse says that the
>> end-comment-marker in inside a string (or inside another comment), we
>> re-parse from the beginning of that string (or comment) to try and see
>> if that end-comment-marker could be considered to close a comment nested
>> within the string (or the other comment).
> That indeed sounds complex.

Actually, it's very straightforward: the forward parse already gives us
the beginning of the surrounding element, so we just re-do the forward
parse from that spot.  It's just a matter of wrapping the code inside
a loop.

>> Calling syntax-ppss every time back_comment is invoked would probably
>> result in bad performance currently: when parsing backward
>> (e.g. backward-sexp), the syntax-ppss-last optimization is ineffective,
>> so we'd fallback on syntax-ppss-cache which ends up scanning on the
>> average syntax-ppss-max-span/2 (i.e. 10K) chars.  When \n is a comment
>> ender (i.e. in most programming language modes), it would imply
>> a forward scan of 10K for every line.

> You're probably right, but I wonder what the benchmarks would say.

> (parse-partial-sexp 1 10000) takes 0.0005 seconds here, so it'd still
> require some intensive usage to show up on user's radar.

> Previously, we started from the beginning of the current defun, as
> delineated by an open paren in the first column, right?

No.  "Previously", we typically scan the line backward and stop as soon
as we hit the previous \n (which tells us that no comment can start
earlier than that if it finishes with a \n).

In a few cases, we do fallback on the forward parse code, in which case
indeed we'll take longer, but those are normally rare (which is why this
comment-cache and my syntax-ppss-patch haven't been installed yet: they
improve the performance of a case that's somewhat infrequent).

> Perhaps we could use the "generic comment bounds" syntax-table property to
> delineate such difficult comments. If that idea sounds similar to
> comment-cache, that is no accident.

Maybe.  Obviously, my syntax-ppss hammer makes me think that such
alternate solutions aren't needed: syntax-ppss solves this case without
having to try and come out with a clever way to detect which comments
are tricky nor how to mark them.

> I've only recently come to the realization that our usage of the
> syntax-table text property has the same general incompatibility with mixed
> mode buffers as comment-cache does. The only reasons why it doesn't show as
> much is because we use them relatively rarely. But we couldn't, for
> instance, apply a "generic string" syntax to some literal in a subregion
> that is inside a "generic string" belonging to the primary major mode.

Indeed.

> Not sure what to do about that.

Not completely sure either.  I've had vague ideas of adding some kind of
hook to syntax-tables, i.e. add a new kind of syntax element which ends
up calling an Elisp function of your choice so you can make it "do the
right thing" for the particular construct.

So when scanning (forward or backward), if we bump into an element with
that syntax (typically applied as a syntax-table text-property), we call
the function which will know how to jump over the sub-region or will
signal an "end of sub-region" error.


        Stefan