Re: forward-comment and syntax-ppss

[Dmitry Gutov]

On 16.12.2016 22:06, Alan Mackenzie wrote:

> I thought the main idea of the facility was to provide a base point for
> a sub-buffer in multi-major-mode modes.  In this case, the code calling
> syntax-ppss isn't going to be using positions less than
> new-local-variable anyway, it's going to be working entirely inside the
> pertinent sub-buffer.

Subregion, you mean? (Same buffer).

But why? Because the multi-major-mode would narrow around the subregion 
as well? If we're considering chiefly this use case, it would make sense 
to just look at the lower narrowing bound, instead of the new variable. 
That would require less code all around.

> The new variable wouldn't have an "arbitrary" (i.e. random) value.  It
> would be carefully set by code which knows what it's doing. I can't see
> how it would be easier (accidentally) to produce an invalid position for
> syntax-ppss.

First, let's remember this.

Second, the code that sets the variable and the code that calls 
syntax-ppss will be different. Typically, they will reside in different 
packages, and they might be authored by different people.

Imagine that our framework code performed the narrowing, then set the 
variable to the lower bound of the narrowing, and then called some major 
mode's fontification code which (for some reason) called widen and then 
syntax-ppss. In this case, it can supply a lower value to syntax-ppss 
than the original narrowing's lower bound.

This situation is a typical problematic one, and a bad error message can 
make it more difficult to investigate.

>  I can't remember any examples of what you mean
> by "the user's intent ... not being known".  Can you give such an
> example of how code would do things differently on the basis of
> "knowing" what the "user meant"?  And I don't mean here the misuse of
> (point-min) to mean "Position 1".  That is just a bug.

For instance, c-indent-line could ignore the narrowing when it was made 
interactively by the user, as typically used by users such as yourself.

But when narrowing is made programmatically, by the multi-mode framework 
code, c-indent-line would honor it.

Also see the Info example in the other email. Having two types of 
narrowing could help fix that scenario.

> > It can be determined by the narrowing (i.e. a function named
> > `inside-a-string?' could confidently know whether to rely on the current
> > narrowing or ignore it, to produce its result, if narrowing were
> > annotated with intent).
>
> Why would anybody need such a facility?  Could you give a concrete use
> case, please?

font-lock-syntactic-face-function normally looks at the current syntax 
state (string, comment, etc).

In the multiple-mode framework, we want to support subregions inside 
comments or strings, and to highlight them accordingly. E.g. code 
example inside a comment, or SQL inside a heredoc literal.

> Narrowing is such a fundamental feature of Emacs that attempting to
> supersede it by something else would be bound to fail, or at least be
> very painful for a long time.  Maybe your invisibility overlays might
> cover some use cases of narrowing, but they would be bound to fail on
> others.

The idea is to move the user-facing narrowing commands to some other 
foundation (like the invisible and maybe intangible text properties), 
freeing the narrowing facility for use by the Lisp code.

> It would foul up much existing code.  It would be bound to.  There will
> be code around which depends on widening meaning exactly "widen".  If you
> change the meaning of `widen' to `(narrow-to-region some-start some-end)'
> things will go wrong.

So far I don't have reasons to believe that it would foul up any use 
cases that worked fine until now.

> Of course, the devil would be in
> the details, but I do think the Emacs team has enough experience of
> changes to be able to do this one in a non-destructive fashion.

Yes, by not making a mountain out of a molehill and just changing the 
existing function. Like the Emacs team has opined already.

> Don't
> forget, the current syntax-ppss doesn't and can't work properly.

Are you trying to hypnotize the readers?

> `syntax-ppss-1' would be usable in the same way as
> `syntax-ppss', so a backward compatibility alias would leave things
> working at least as well as they do at the moment.

Migrating all the code out there to use the new function is a big job by 
itself, and it would spread over years. That's why we rarely do 
something like that.

Logistics aside, either they are different, or they are the same. If 
there are no use cases for syntax-ppss in which it can't be replaced 
with syntax-ppss-1, there's literally zero reasons to create a new 
function instead of fixing the existing one.

> That was precisely the mechanism I was looking at a few days ago.  I
> agree it could probably be improved upon, perhaps after gaining
> experience with it.

I'm saying it can't be perfect without taking into account the calling 
code. Leaving all cache management to it is one way to do that (but not 
the only one, to be clear).

But if you'd like to gain some personal experience with it, you're 
welcome to make the planned changes in your copy of Emacs, download 
mmm-mode, adapt it accordingly (there are just a few places in 
mmm-region.el, all calling narrow-to-region), and play around with it.

> > With syntax-ppss-dont-widen, the caller code can take the responsibility
> > of maintaining the values of syntax-ppss-cache and syntax-ppss-last.
>
> The calling code shouldn't even be aware of the internal details of the
> syntax-ppss cache, far less be burdened with the responsibility of
> maintaining it.

That's not a given. Like you said above, the caller code will have to 
"knows what it's doing". A multiple-mode framework will have to be 
familiar with internal structure of a number of packages anyway, at 
least to some extent.

> Such high coupling between syntax-ppss and its users
> would make it pretty much impossible to change the cache mechanism at
> a later date if we wanted to do that.  syntax-ppss should be as
> black-box like as possible.

If we change the cache mechanism, we can adapt these special callers to 
it. Yes, normally, we want to limit the public API in order to improve 
the stability of the ecosystem.

In this case, however, there are only 2-3 active packages that work in 
the "multiple-mode" space. And there are unlikely to crow up many more 
of them because it's a hard problem.

Even if I'm wrong, and by some miracle (or hard work of improving our 
internals, of course) many new packages like that start to appear, we'll 
have time to choose a stable interface and update the current multi-mode 
packages. In the meantime, syntax-ppss-dont-widen's documentation will 
just have to be clear that any packages using it are on their own.

> > In short, letting the caller code control cache invalidation has its
> > benefits.
>
> They are by far outweighed by the disadvantages.

While I don't agree, you've pointed out the disadvantage of the simplest 
approach to caller-controlled cache invalidation. There can be others. 
Maybe a macro like `with-separate-syntax-ppss-cache'.