Re: Initial fontification in sh-mode with tree-sittter

[Yuan Fu]

> On Nov 2, 2022, at 12:17 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: João Paulo Labegalini de Carvalho <jaopaulolc@gmail.com>
>> Date: Wed, 2 Nov 2022 12:34:56 -0600
>> 
>> Yes, the fontification in the buffer is still not updated correctly.
>> 
>> With the step below, the definition of function foo is not re-fontified and 
>> remains with string-face until C-x x f is
>> executed.
>> 
>> emacs -Q from top of the feature/tree-sitter branch
>> M-: (require 'treesit)
>> M-x customize-variable treesit-settings RET
>> Set "Activate" to "Yes" and apply the change.
>> C-x b sample.py RET
>> M-x python-mode
>> Write the following program:
>> 
>> def main():
>>    return 0
>> 
>> M-<
>> """
>> M->
>> """ (at this point everything is in string-face, as expected)
>> C-a
>> C-k (everything is still fontified as string)
>> C-x x f (leading """ is not fontified and definition of foo is correctly 
>> fontified)
>> 
>> I am interested in understanding what is causing this as a similar thing 
>> happens with heredocs in sh-mode &
>> tree-sitter.
> 
> Yuan, can you look into this?  It sounds like some general problem
> with integration of tree-sitter with jit-lock.  

Yeah, this is what I’ve been working in the past two days. I just pushed a 
change f331be1f074d68e7e5cdbac324419e07c186492a which should fix it. [ I just 
saw that I missed a function in the commit message, I guess I can’t fix it now 
:-( ]

If you are interested to know what’s going on:

The problem is as I described earlier, when the user inserts the starting quote 
(“””), the parse tree is incomplete and there is no string node. Only when the 
user inserts the ending quote is there now a string node to be captured by the 
fontification rules. 

For example, in this snippet there are two regions A and B

"""          ---+
def main():     | Region A
    return 0 ---+
             ---+ 
"""             | Region B
             ---+

when user inserts the “”” in B, and jit-lock fontifies region B, it captures 
the string node and the part of the string in region A needs to be updated. If 
we fontify the whole string in string face, redisplay does not immediately 
reflect the change in region A (maybe due to some optimization? jit-lock is 
definitely aware of this, see jit-lock-force-redisplay).

Redisplay not reflecting the change in face is just the surface problem, and 
can be fixed by setting fontified to t on region A, which seems to trigger 
redisplay to reflect changes immediately (this is what jit-lock-force-redisplay 
does). The deeper problem is, if there is some regex-based-font-lock face in 
region A (applied when Emacs fontified region A), eg, highlighted TODO keywords 
in a docstring, they will be overwritten by the string face, if we just apply 
string face to the whole string and trigger redisplay.

Maybe we can apply string face and re-run regex-based font-lock on the whole 
string, but that works against jit-lock. If the string is long regexp-font-lock 
might take a long time.

What I ended up doing is to set jit-lock-context-unfontify-pos to the beginning 
of the string node (aka beginning of region A). Then in a timer 
jit-lock-context will refontify everything after that position. And I have some 
measure to break possible infinite recursion (fontify region -> set 
jit-lock-context-unfontify-pos -> cause refontification -> fontify region -> …).

The alternative, where we put everything in string face when we see an opening 
“””, is not really feasible. It’s kind of feasible in python, where an opening 
“”” alone looks like (ERROR “””), aka the quote node inside an error node. But 
if you insert /* in javascript, you get (ERROR “/“) and (ERROR 
(regexp_pattern))—tree-sitter can’t tell that they are opening comment (which 
is fair). Plus this approach requires non-trivial involvement from each major 
mode: each major mode needs to somehow tell Emacs what is a “potential opening 
comment/string”.

> Or maybe something is
> missing from the way tree-sitter nodes are mapped into face text
> properties.  Are the faces actually being put on the relevant text,
> for starters?

They are. Fortunately this is not the problem.

Yuan