Re: Bug #22983 (syntax-ppss returns wrong result) is still open. Could we fix it before the release, please.

[Alan Mackenzie]

Hello, Dmitry.

On Mon, Jun 13, 2016 at 01:58:02AM +0300, Dmitry Gutov wrote:
> On 06/12/2016 12:34 PM, Alan Mackenzie wrote:

> > I can't reason about hard-widen very much, because I haven't read its
> > specification.

> The change to the narrow-to-region's docstring in the patch contains the 
> specification.

It does not.  It contains a few relatively vague aspects, undefined
terms, and "for full details, search!".  I'm not trying to slag off
Vitalie here, because clearly he was in the middle of trying to figure
things out, and was unclear about things.

> Here's a better link to the message:

> http://lists.gnu.org/archive/html/emacs-devel/2016-03/msg01576.html

I've glanced through that, and I found the doc string you've referred
to.  It's some help.

> > No.  A careful consideration of its proposed working before
> > implementation (let's call it a design ;-) can reveal a lot about how
> > well it could work.

> ...and time, if we're being realistic.

> > `widen' makes the entire buffer accessible.  `narrow-to-region' makes
> > the specified region the accessible portion.  And that's it!  What could
> > be simpler?

> If you take a shovel and remove its blade, the result will look very 
> simple. That doesn't mean it's appropriate for any interesting task.

So, you consider widen and narrow-to-region in their current form as
being as useless as a detached shovel blade?  I think you're just being
perverse, here.  widen and narrow-to-region are enormously useful, and
it is their simplicity, in large part, which makes them so useful.  You
are proposing to destroy this simplicity.

> Nowhere does narrow-to-region's documentation say it should only be used 
> in major modes.

Of course not.  As I've said, it's a general purpose tool used
univerally in Emacs.  I would like to keep it that way.

> > I have encountered no such problems in over 15 years of hacking Emacs.
> > I think you're seeing problems somewhere and blaming
> > narrow-to-region/widen, when the real problem is somewhere else.

> It's been a known problem for a while, and it has come up in multiple 
> discussions over the last years.

"It's been known"?  To whom?  The only discussion I have seen on the
topic has been between Stefan, yourself, and Vitalie.  Apologies if I've
missed anything.

> The "real problem" is a matter of perspective. Some might argue that 
> trying to use narrowing at all, for any purpose, is a bad idea, ....

That's plain ludicrous, given the number of couterexamples in Emacs.

> .... and then, of course, the right choice would be to use something
> else.

You seem to want to make widen/narrow-to-region useless for some things.
That is the crux of the matter.  Before making any such change, I'd
rather know what capabilities are being discard in the complexifying of
these primitives.

The right choice for multiple major modes might well be to use something
other than narrowing.  As you know, I proposed something else a few
weeks ago.

> >> How else would you apply each major mode's fontifications only within
> >> its subregions?

> > By adding the appropriate structures as buffer local variables (or
> > perhaps as text properties) which would delimit the subregions, then
> > enhance Font Lock to respect those boundaries.

> Please go ahead and write up this proposal in detail.

No, I was simply answering your question, giving you ideas for further
ideas.

> It doesn't seem as simple as you make it sound. And I wonder if it
> ends up to be different from the "islands" proposal.

Nothing we do here is going to be simple, admittedly.  There might well
be a way of doing things which is a bit like the "islands" proposal but
without its main disadvantage (the huge effort to implement it), and
also a bit like the complexification of narrowing, but without its
disadvantages.

> Further, font-lock is not the only facility we want here. Making 
> syntax-ppss obey region boundaries is another. There's also indentation 
> code, which is more difficult to handle, but limiting visibility there 
> somehow also seems essential.

OK.

> The super mode can also adapt different simpler facilities, like Imenu, 
> using the same approach (go over the chunks, narrow to each, scan using 
> the language-appropriate tools, and concatenate the results).

> >> See
> >> https://github.com/purcell/mmm-mode/blob/c9a857a638701482931ffaaee262b61ce53489f3/mmm-region.el#L789-L816

> > That's rather a lot to take in before breakfast!

> Did you have the breakfast yet? It's just 30 lines.

Ah, I see!  It wasn't clear to me that you were talking just about a
single defun in that file.  I'm not accustomed to reading and
interpreting long URLs.  That was a function which fontifies a number of
regions.  I've forgotton why I should be interested in it.  ;-(

> >> super modes are currently implemented as minor modes. major modes
> >> shouldn't override the choices made by minor modes, if at all possible.

> > What????

> What I said. To the extent the facilities allow, of course.

> > The use/non-use of narrowing is NOT a setting, any more than the
> > use/non-use of cdr is.  They're just general purpose tools.

> Turns out, they are not general purpose enough. That's the problem.

That's an interpretation of the problem.  Your solution appears to be to
make narrowing LESS general, specific to a single problem.

> >> There have been several discussions.

> > They're not coherent, and they're not very useful.

> If there's anything in them that makes you confused, I've yet to hear 
> about it. As far as I can see, you're not interested in making even that 
> effort in learning the subject.

There's nothing in particular that "leaves me confused".  It's that
there no coherent description.  To try and pick everything up from a few
discussions on emacs-devel going arbitrarily far back would take me
several days, and even then I couldn't be sure I'd not missed anything.

> > Could you please write this document.  Given the massive change you want
> > to make in a fundamental Emacs tool, this is not unreasonable to ask.

> I don't enjoy rehashing others' arguments when the original messages are 
> readable enough.

The original messages aren't readable enough for this purpose; some
things are said many times - other things are left out altogether.  The
whole discussion is an incoherent jumble.  That's the nature of mailing
list discussions.  Why do you think I put so much effort into getting
the "islands" proposal coherent - everything said once, and as far as
the flow of the ideas would allow, only once?

What I'm not asking you to rehash is other people's arguments - only the
salient technical points, "one per line".

> It's a small change, by itself. Please read the discussion, starting 
> with http://lists.gnu.org/archive/html/emacs-devel/2016-03/msg01129.html

> And http://lists.gnu.org/archive/html/emacs-devel/2016-03/msg01182.html 
> in particular. After that, please let me know if you have any specific 
> questions.

One of these has a patch, which looks fairly simple.  One thing which
isn't simple is replacing

    (widen)

with

    (let (hard-widen-limits)
      (widen))

in all the places needed.  (I've counted ~705 uses of `widen' in our C
and Lisp sources.)

> > This is, after all, what I did for the "islands" concept a couple of
> > months ago.

> And I have spent considerable time contributing to that discussion as well.

Which is appreciated.

> >> The main thing CC Mode would have to worry about from then on, is that
> >> it won't always be able to goto-char to positions beyond the hard
> >> narrowing, even if they exist in the buffer (and are pointed to by some
> >> buffer-local data structure maintained by CC Mode's parser).

> > That sounds like something worthwhile sorting out in advance.

> Sure. Please go ahead and present your thoughts on the subject.

I can't, as yet.  I don't have enough information.

> I realize it might be rather complicated, actually, to use the narrowing 
> approach with CC Mode as submode. But I don't see any alternative 
> solutions that we're likely to implement. So it may be that some modes 
> will still refuse to work in multi-mode context.

Maybe that design document might, just might, help sort out such
difficulties before they occur.  What I foresee is that these modes
won't flat-out refuse to work with multi major modes, they'll just have
nasty to debug bugs.

And maybe, just maybe, we might somehow get back to discussing the
actual topic of this thread which is a bug concerning syntax-ppss.  One
of the consequences of coupling functions which need not be coupled is
that discussions like this can veer far off topic.

-- 
Alan Mackenzie (Nuremberg, Germany).