bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.

[Alan Mackenzie]

Hello, Stefan.

On Fri, Mar 18, 2016 at 12:49:07AM -0400, Stefan Monnier wrote:
> > Do this by adding two new fields to the parser state: the syntax of the last
> > character scanned, and the last end of comment scanned.  This should make 
> > the
> > parser state complete.

> Thanks.  I like the "syntax of the last character scanned", but I don't
> understand the reasoning behind "last end of comment scanned".  Why is
> this relevant?  Is it in case the "last character scanned" was a "slash
> ending a comment" so as to avoid treating "*/*" as both a comment closer and
> a subsequent opener?

That's exactly the reason.

> If so, I'm not sure I like it.

I don't really like it either.

> It sounds to me like there's a chance it's actually incomplete (e.g.
> it doesn't address the similar problem when the "last character
> scanned" is an end of a string which also happens to be a valid
> first-char of a comment-starter), and even if it isn't, it "feels
> ad-hoc" to me.

Now even I wouldn't have come up with that end-of-string scenario.  ;-)
Such a scenario is presumably one reason why, in scan_sexps_forward, two
character comment delimiters are handled before strings.

> Would it be difficult to do the following instead:
> - get rid of element 11.

Done.

> - change element 10 so it's nil if the last char was an "end of
>   something".  Another way to look at it, is that the element 10 should
>   only be non-nil if the "next lexeme" might start on that
>   previous character.

I've tried this, and it's somewhat ugly.  Setting the "previous_syntax"
to nil is also needed for the asterisk in "/*".  The nil would appear to
mean "the syntactic value of the last character has already been used
up".  So the "previous_syntax" is nil in the most interesting cases.  It
also feels somewhat ad-hoc.

How about this idea: element 10 will record the syntax of the previous
character ONLY when it is potentially the first character of a two
character comment delimiter, otherwise it'll be nil.  At least that's
being honest about what the thing's being used for.

> I also have a side question: IIUC your patch makes the 5th element
> redundant (can be replaced with a test whether "last char syntax" was
> "escape"), is that right?

It would appear to be, yes.  We really can't get rid of element 5,
though, because there will surely be code out there that uses it.  But
if I change element 10 as outlined above, element 5 will no longer be
redundant.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).