Re: A vision for multiple major modes: some design notes

[Alan Mackenzie]

Hello, Eli.

On Sat, Apr 23, 2016 at 10:39:55AM +0300, Eli Zaretskii wrote:
> > Date: Fri, 22 Apr 2016 22:35:08 +0000
> > Cc: address@hidden, address@hidden
> > From: Alan Mackenzie <address@hidden>

[ .... ]

> Please let's not forget that regexps are used in many places that have
> no relation whatsoever to major modes, and searching for whitespace is
> a very common operation using regular expressions.  Infecting all
> those with this new meaning of whitespace that is totally alien to any
> code that doesn't deal with major mode is IMO plain wrong.

> More generally, I think we should first and foremost make our goal to
> have a clean and reasonably simple design, and only care about the
> amount of changes in major mode code as a secondary goal.  Thinking
> about the changes in major modes first could easily lead us astray.

We must consider both these things together.  A prime design goal is to
allow an arbitrary major mode to be used by a super mode with the minimum
of adaptation to the major mode, ideally none.

> > Bear in mind that this matching of an island by a whitespace regexp
> > element would happen ONLY whilst `in-islands' was bound to non-nil, i.e.
> > when a major mode is working in its own island chain.

> I understand, but I don't think this goes far enough to address my
> concerns.  And my suggestion to have a separate class/category will
> serve your needs just as well, so I'm unsure why we need to piggyback
> [:space:].

If this new category (say, "[:gap:]") only needed to be used in super
modes or subsystems, I might agree with you.  But if [:gap:] needs to be
used in major mode code, that involves massive amounts of editing, which
would make the new mechanism much less useful.

> > Are there any circumstances in which we would not want the major
> > mode to see the gap between its islands as WS?

> Who says that every major mode necessarily treats whitespace as you
> assume?  Most (or even all) of those you know about might, but this is
> not written anywhere as a limitation of a major mode.

It will become expected of a major mode that it doesn't tamper with stuff
outside of its own island chain(s).  That would violate the abstraction
that the island mechanism represents.

> By hard-wiring this special meaning of [:space:] into your design, you
> are limiting future (and possibly some rare extant) major modes.

I don't think it's all that special.  It's natural.  Ideally, a major
mode should see its island chain as the whole buffer.  It should be
unaware that it is running in an island at all.  I see this treatment of
a :gap: as whitespace as the most natural way of implementing this.

Major modes which would violate the island abstraction, if there are any,
shouldn't be used in this new multi major mode mechanism.

> > When `in-islands' is nil (i.e. when the super mode's code is
> > running, or the user is typing commands) the islands would NOT match
> > a WS regexp.

> Are you sure that none of the background processing will ever need to
> treat islands as such?  I'm talking about stuff like timers, process
> filters and sentinels, hook functions run by redisplay and the command
> loop, etc.

All these subsystems will need to be aware of whether they are dealing
with the buffer as a whole, or merely with an island chain.  They will
need to bind `in-islands' appropriately, frequently using the value that
was current when they were invoked.

> If any of these might need to observe the island rules and
> restrictions, the design which builds on in-islands being bound to
> non-nil _only_ when the major mode is running its own code is
> unreliable, and will cause unrelated code to find itself dealing with
> island peculiarities.

Perhaps I expressed myself too forcefully and literally.  If any of these
things are dealing with an island as a unit, they are surely "running
major mode code" in this sense.  Clearly things like redisplay and font
lock will need explicitly to set and clear `in-islands' when they are
dealing with island chain stuff.

> E.g., JIT font-lock runs off an idle timer, but clearly needs to
> observe islands, so it sounds like the problem I'm worried about is
> pretty much into our faces.

The font-lock and jit-lock entry points will need to set `in-islands'.

> > > By contrast, if we decide that whitespace matches an island, we are
> > > opening a giant can of worms.  Here's one worm out of that can: some
> > > low-level operations need to search the buffer using regexps
> > > disregarding any narrowing -- what you suggest means these operations
> > > cannot safely use whitespace in their regexps.  This is something to
> > > stay away of, IMO.

> > It depends on whether these low level operations are working within an
> > island chain (`in-islands' non-nil) or on the buffer as a whole
> > (`in-islands' nil).  I think such operations would typically be run with
> > `in-islands' nil, hence would not run up against these problems.

> "Typically" is not good enough, IMO.  We must convince ourselves that
> this happens _always_, and there will _never_ be a reasonably
> justifiable need to search the entire buffer for whitespace when
> in-islands is non-nil, i.e. in any of the code that is running as a
> side-effect of performing some major-mode related operation.

I agree with that.

[ .... ]

> > > If it is stored in the text property, then you will have to decide
> > > what happens when text is copied and yanked elsewhere.

> > It would be the job of the `island-after-change-function' to strip the
> > unwanted text properties (both the `island' and `syntax-table' ones) and
> > to apply any needed new ones to the yanked region.

> The problem is the decision whether they are unwanted or not.  It's
> usually not simple to make that decision for text properties that
> change the way text is displayed, when surrounding text also affects
> that.

But that decision has to made somewhere, somehow, by the super mode,
regardless of how multiple major modes are implemented.  Just for
clarity, `island-after-change-function' is a hook, not a fixed function,
and writing a super mode's function for this hook would be a substantial
part of writing that mode.

[ .... ]

> > #define CVAR(var, buf, position) \
> >     chain = read_text_property (Qisland, buf, position), \
> >     chain ? chain.var \
> >           : BVAR (var, buf)

> > , but I don't think that would be a valid Lvalue in C.  :-(

> Didn't you talk about some alist to look up?  I see no alist look up
> in this pseudo-code.  And 'chain.var' sounds wrong, since 'chain' is
> definitely a Lisp object, not a C struct.  Or maybe I don't understand
> what hides behind read_text_property.

I think we're talking at cross purposes.  I'm proposing that CVAR would
take the place of BVAR for many of the variables which are slots in the
struct buffer, and they would also become slots in the new struct chain.
The other variables, currently held in `local_var_alist_' in struct
buffer, would be accessed from a `local_var_alist_' element of the struct
chain in much the same way.

Or perhaps there is a better way of implementing chain local variables.
It is more of an implementation detail than an essential part of the
design.

[ .... ]

> > I'm not at all clear on when and how buffer local variable
> > bindings get swapped in and out of, say, C variables like Vfoo.

> This happens when we switch buffers, see set_buffer_internal_1.

Thanks!  I see that now.  I had spent some time looking for that code in
data.c.

> But that function is driven by an explicit event of switching buffers,
> while in your design you need to do something similar when point
> crosses some buffer position, which is a much more subtle event.  E.g.,
> think about all the save-excursion and save-restriction code out there.

A good point.

[ .... ]

> > > What is "chain locality"?

> > Having things (variables) which are local to a chain, as opposed to
> > global variables or buffer local variables or frame local variables.

> OK, but no one said that applying a restriction and making
> island-specific bindings of variables must be parts of the same
> feature.  They could be 2 separate features instead.

Maybe they could.  Maybe it wouldn't make much sense.  I'd have to think
about that a bit more.

[ .... ]

> > Surely a similar mechanism [ to swapping buffer local bindings into
> > and out of fixed variables in the C code ] could be created for when
> > the current island changes.

> The issue is to make it as cheap as possible, because redisplay code
> is at liberty to move around the buffer at will, and the location
> where it examines buffer text is not directly related to point.

Yes, this would require careful design and coding.  One detail struck me
immediately on seeing the code in set_buffer_internal_1.  The code has to
cdr its way down the entire list of variables in local_var_alist_,
despite the fact that only a few of them point to C variables.  Maybe it
would make sense to extract this smaller list into a separate chain.

[ .... ]

> See above: there might be some situations, like JIT font-lock, where
> you will want to have in-islands non-nil while running async code, and
> that might make the islands visible to code that is not strictly part
> of any major mode, like the infrastructure which invokes these async
> parts of Emacs code.  So I think you need to consider the effects of
> those on more than just major modes.

Yes, indeed.  The challenge here will be to identify all of the pertinent
subsystems.

-- 
Alan Mackenzie (Nuremberg, Germany).