Re: syntax-propertize-function vs indentation lexer

[Stephen Leake]

Stefan Monnier <address@hidden> writes:

>>> Sounds expensive.  How does it cope with large buffers?
>
>> Not clear yet - I'm still getting the Ada grammar right. 
>
>> The parser is actually generalized LALR, which spawns parallel parsers
>> for grammar conflicts and ambiguities. So it can be very slow when the
>> grammar has too many conflicts or is ambiguous - running 64 parsers in
>> parallel is a lot slower than running 1 :). But it works well when the
>> conflict can be resolved in a few tokens, and is much easier than
>> reconstructing the grammar to eliminate the conflict.
>
> Aha!  So the parser is a separate executable written in some other
> language than Elisp, I guess?  In that case parsing speed should not be
> a serious concern (except for the possible explosion of parallel
> parsers).

No, it's in elisp. See
http://stephe-leake.org/emacs/ada-mode/emacs-ada-mode.html for the code.

> Although, using it for indentation makes speed a real concern:
> in many cases one does "edit+reindent", so if you put a "full reparse"
> between the two, it needs to be about as fast as instantaneous.

That's how my current SMIE-based parser works, and it's "fast enough".

I'm working on replacing it with an LALR parser, because the resulting
code is much cleaner.

>> Such a file would never be accepted in any project I have worked on, in
>> any source language, unless it was generated from some other source.
>
> I agree that 1MB is very unusual, but emacs/src/xdisp.c is pretty damn
> close to 1MB.  And I've seen several times files of several hundred
> KB.

Ok.

>> I'm also using the cached data for navigation (moving from 'if' to
>> 'then' to 'elsif' to 'end if' etc); that is logically independent of
>> indentation (but not of the parser, of course).
>
> Then navigation should also call syntax-propertize (indeed smie's sexp
> navigation also calls syntax-propertize for the same reason).

Yes, I think this is the best solution.

>> Since the parser is asynchronous from the indentation, it would have to
>> go in the parser (actually lexer) code. wisi-forward-token would be a
>> logical place. But what would be the right guess for 'end'? The first
>> step in wisi-forward-token is forward-comment, which can skip quite large
>> portions of the buffer. 
>
> I have the same problem in SMIE navigation, indeed.  For backward
> navigation, that's not a problem, but for forward navigation, I don't
> have a good answer.  Luckily, SMIE mostly cares about backward
> navigation since that's what needed for indentation, but currently
> forward navigation can bump into parse bugs for failure of calling
> syntax-propertize on the text being considered.

In my case, putting the call to syntax-propertize in wisi-parse-buffer,
not in wisi-forward-token, solves the problem; wisi-parse-buffer always
parses the whole buffer :). This could easily be generalized to take an
'end' arg.

>> How does syntax.el take care of this? The only function on
>> after-change-functions by default is jit-lock-after-change. And that's
>> only there if font-lock is on.
>
> It's added to before-change-functions.

Doh!

Thanks,

-- 
-- Stephe