bug#73404: 30.0.50; [forward/kill/etc]-sexp commands do not behave as expected in tree-sitter modes

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  Eli Zaretskii
>  <eliz@gnu.org>,  Mickey Petersen <mickey@masteringemacs.org>,
>   73404@debbugs.gnu.org,  Theodor Thornhill <theo@thornhill.no>
> Date: Thu, 12 Dec 2024 09:17:44 +0200
> 
> >>> Ah, this matches my idea of defining sexp in other languages as 
> >>> “repeatable
> >>> construct/list-like construct”.  We went with “every syntactic construct” 
> >>> at
> >>> the time, which I didn’t object to, but I’m definitely happier with the
> >>> repeatable construct approach. Including Stefan and Theo since they were
> >>> part of the original sexp navigation discussion.
> >> 
> >> FWIW, we have both `forward-list` and `forward-list` and the new
> >> behavior you suggest sounds closer to the historical behavior of
> >> `forward-list` than `forward-sexp`.
> >
> > Actually, what’s the difference between forward-list and forward-sexp?
> > I always thought they are the same at least for Lisp.
> 
> forward-sexp moves over a balanced parenthetical group like
> forward-list does.  Plus forward-sexp also moves over an atom
> such as a symbol, a number.
> 
> The problem is that treesit adds too much structural information
> to such simple things as a symbol and a number.  For example, in js
> a simple keyword "export" gets the "(export_statement export" subtree,
> Another keyword "const" gets "(lexical_declaration kind: const", etc.
> 
> Therefore for such symbols forward-sexp needs to bypass the structure
> and use simpler syntactic information to move over them like on a flat list.

If you mean we should ignore the information provided by tree-sitter
and instead use our own syntactic information, then that sounds wrong
to me, FWIW.  Why cannot we understand enough of the tree-sitter
structural information to move like we want?  Presumably, the
structural information provided by tree-sitter is a portion of a parse
tree, which to me means we should be able to move between the parse
tree's nodes as long as we understand the tree and can interpret it in
our terms.

Aren't there some grammar-agnostic traits of tree-sitter nodes that
would allow us to interpret the nodes in language-independent terms?
If that is not available, then each major mode will have to provide
treesit.el with a way to interpret the tree-sitter nodes of the
corresponding grammar in a way that will allow sexp movement, thus
providing an abstraction layer that treesit.el could use for the
movement commands.