bug#73404: 30.0.50; [forward/kill/etc]-sexp commands do not behave as expected in tree-sitter modes

[Juri Linkov]

>>> Gentle reminder that `forward-sexp` is not a "list-navigation" function.
>>> That would be `forward-list`.  We very often use sexp commands and
>>> functions to manipulate non-lists such as identifiers.
>> Do you think it would be better to override low-level functions 'scan-lists'
>> and 'scan-sexps' with new variables like 'scan-lists-function'
>> and 'scan-sexps-function', instead of adding more variables for
>> overriding top-level commands such as a new variable 'forward-list-function'
>> and 'down-list-function', like the existing 'forward-sexp-function'?
>
> Don't know.

I tried to override `scan-lists` and `scan-sexps` with advices
(a proof of concept attached below), and it works nicely.
For example, `C-M-d` moves down to the HTML element in html-ts-mode.

But then I realized there are not too many places where such
overriding might be useful.  In fact, there is only 1 place
in `show-paren--default` that uses `scan-sexps`.  But even this
occurrence can't be used, so I created `treesit-show-paren-data`
for `show-paren-data-function` in bug#75122.

And overriding `scan-lists` is useful only in `forward-list`,
`down-list` and `up-list`.  That's all.  So clearly instead
of overriding `scan-lists` and `scan-sexps`, better would be
to add 3 new variables: `forward-list-function`,
`down-list-function` and `up-list-function`.

This gives the users more flexibility to choose for example navigation
for C-M-f with a limited number of treesit lists + syntax symbols, and
for C-M-n everything that is a list in treesit.  This means that with

  start_atimer (-!-enum atimer_type type, struct timespec timestamp)

C-M-f could skip only the next symbol and move to

  start_atimer (enum-!- atimer_type type, struct timespec timestamp)

while C-M-n could skip the whole next `parameter_declaration`

  start_atimer (enum atimer_type type-!-, struct timespec timestamp)

or vice versa depending on the values of all new options `...-function`
in ts-modes.

> What I do know is that in general we'd also want an `up-sexp` operation.
> Currently we have an ugly kludge in `up-list` to try and use
> `forward-sexp-function` (which is ugly both because
> `forward-sexp-function` doesn't really provide the functionality we
> need, and because it mixes up sexp and list navigation), and it would be
> good to clean it up.

Agreed, this distinction is required for treesit.  Hopefully,
this can be achieved by separating `forward-sexp-function`
and `up-list-function` that in ts-modes could be set to new
functions either `treesit-up-list` or `treesit-up-sexp`.

PS: will try to refactor everything in this attachment to new
`forward-list-function`, `down-list-function` and `up-list-function`:

scan-lists-advice.el
Description: application/emacs-lisp