Syntax-propertize and CC-mode [Was: Further CC-mode changes]

[Alan Mackenzie]

Hello again, Stefan.

I've separated the discussion on syntax-propertize from the rest,
because things are threatening to get out of hand.

On Sat, Sep 13, 2014 at 03:24:03PM -0400, Stefan Monnier wrote:

> >> You know why: `syntax-propertize' is used by all major modes except
> >> cc-mode derived ones.  And it's more efficient, because it's more lazy.
> > Lazy?  Does C-M-f, for example, trigger syntax-propertize in any way?

> Not yet, no.

OK.  I'm not yet convinced that syntax-propertize works well in languages
like C++ and Java, where template/generic delimiters < and > are marked
with category properties.  (These have symbols with syntax-table
properties.)

Because applying these properties is expensive at runtime, they are tended
to with loving care in CC Mode.  When a buffer change is made, a
before-change function c-before-change-check-<>-operators removes the
category properties from exactly those pairs of <>s which need it, in
particular, the removal region is LIMITED TO THE CURRENT "STATEMENT".  (A
template/generic construct cannot include braces or semicolons.)  The
properties get applied, when necessary, as a side effect of the scanning
which happens during font-locking.  It is a long-standing bug that they
aren't applied when font-locking is disabled.

> >> Also its correctness argument is much simpler since it doesn't rely on
> >> interactions between before-change-functions, after-change-functions
> >> (and font-lock), contrary to your code.
> > The argument could run that because syntax-propertize doesn't take
> > account of these subleties, its actions aren't always correct.  But ...
> > syntax-propertize isn't documented in the Elisp manual.  It's unclear
> > when it gets called (in relation to after-change-functions, for example)
> > and even _if_ it always gets called (e.g. when font-lock isn't enabled).

> That's because it's irrelevant: if *you* need the properties to be
> applied, then call (syntax-propertize POS).  If they've already been
> applied, then this will return immediately.

That's useful to know.  I didn't know that (and couldn't have known it)
before, since it is undocumented.  How is the lower limit determined
(matching the upper limit POS)?  At a guess, this is going to be anywhere
down to BOB.  This isn't very lazy.  Why doesn't syntax-propertize take
TWO parameters, BEG and END?

> Since font-lock's syntactic fontification needs those properties to be
> applied, it does call syntax-propertize.

I didn't know this either.  How will this be interacting with C++/Java
Modes' use of category properties to set the syntax?  I don't have good
feelings about this.  

Looking at the source, syntax-propertize is unoptimised.  It will always
erase the syntax-table properties from the last buffer change onwards, in
contrast to C++ Mode, which only erases them locally as needed.  There is
surely an opportunity for better customisation here.

> Similarly, indent-according-to-mode calls it since it's usually a good
> idea.  And yes, C-M-f should too, tho noone has been sufficiently
> bothered by it to implement it yet.

> > Adapting code to use it is thus a _massive_ amount of work - it involves
> > reading the source code for syntax-propertize and friends to work out
> > what it does.

> No you don't: the contract is just what I stated above and you don't
> need to know more than that.

That is not true.  I need to know that syntax-propertize would nullify
C++/Java Modes' careful optimisations.  I need to know whether or not
syntax-propertize properly handles category properties.  I suspect it
doesn't.  That involves reading the source code, for lack of
documentation.

> > I suspect that for a mode which needs syntax-table text properties
> > early in its after-change-functions, syntax-properties wouldn't work.

> Suspect all you want.  But I assure you it works just fine.

> >> > It's a lot of work to change,
> >> It's work I did, pro bono.  Just enjoy it.
> > No, you haven't.  What's the equivalent of `syntax-propertize' in XEmacs,
> > or an old GNU Emacs?

> Did you look at my patch?  It only uses syntax-propertize
> when available.  When not, it keeps using your old code.  And the bulk of
> the code is shared between the two cases.

I looked at the patch, yes, but didn't catch every last detail.  There's
the philosophical question why add this extra code when it's not
replacing old code, but causing bloat.  I now think syntactic-propertize
contains a severe pessimisation: At a buffer change, all syntax-table
properties are effectively erased between the change point and EOB.  This
wasn't done in the existing AWK Mode code.  Perhaps it doesn't matter too
much, since AWK buffers tend not to be very big.

> > Right.  So I'm now going to have three alternative code flows in place of
> > the one I currently have: (i) One flow for when font lock's enabled; (ii)
> > one for when FL is disabled, which explicitly calls syntax-propertize
> > from after-change-functions; (iii) one for Emacsen lacking
> > syntax-propertize.  I've already got (iii).  I think you're asking me to
> > add (i) and (ii).  This is an increase in complexity and a lot of work.

> There are many different ways to handle backward compatibility.  You can
> do it as above (where my position would be to simply keep the current
> code for the old Emacsen that don't have syntax-propertize, so while
> there are more different ways to work than you'd like the old ways have
> been tested and there's no reason they should stop working, so you can
> largely ignore them).
> Or you can provide you own implementation of syntax-propertize.
> It's actually simple.  If you prefer this route, I can do
> that as well.

syntax-propertize needs a way of customizing its before-change function
to avoid removing ST properties needlessly from large chunks of buffer.
How about `syntax-depropertize-function', which if non-nil would be run
in place of a bit of syntax-ppss-flush-cache?  Or something like that?

[ .... ]

> syntax-propertize uses a before-change-function to "flush the cache", of
> course.  After that you just call it when you need it.  E.g. If
> font-lock itself doesn't call it, then we'll need to arrange to run it
> before font-lock does its thing.

[ .... ]

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).