Re: [Emacs-diffs] emacs-25 f8208b6: Document the user-level features of the Xref package

[Dmitry Gutov]

On 01/10/2016 06:50 PM, Eli Zaretskii wrote:

> Sorry, I don't understand: AFAIK (and you seem to confirm) we do use
> these tools in some situations, so the text is correct in saying that
> they are used when appropriate.

No, these are programs that we just use in some circumstances.

The text says: "Each such tool is used as a @dfn{backend} by commands 
described in this section". This is untrue, they're not used as 
"backends" in Xref terms. And most of them are only used by one command 
(singular, not plural): xref-find-references, and only when an actual 
backend doesn't provide a replacement logic.

> Is it that I used "backend" in a meaning
> that's different from what the xref implementation uses?

Yes. You can call them "programs", or "external tools" like CEDET's docs 
prefer to call them.

This part also makes using the term "backend" for them inaccurate: "Each 
command detects which backends are available for the current major mode".

Xref backends *do* depend on the current major mode. Or can depend, at 
least. But whether we use ggtags, cscope, or grep, doesn't depend on the 
current major mode at all, just on whether there is an index file 
generated by the respective external program, in some root directory.

semantic-symref-filepattern-alist uses the major-mode value to derive 
the file wildcards to search for, but not, again the tools to use.

> If so,
> that's deliberate: this is a user manual, and that was the only
> reasonable way I could think of to convey the idea of several backends
> without talking about technicalities like xref-backend-references,
> what symref does, when do we use the fallbacks, what are generic
> functions, etc.

A backend is a value that can be returned by xref-find-backend. By 
diluting the term, you all but make sure that the same users will become 
confused later, when they peek under the covers, to try to fix 
something, or use xref in their own programs, etc.

If you think the users are fine with only superficial understanding, 
let's keep the description at that level, and not include misleading 
specifics.

> The idea the user will hopefully take out of this is that we use
> several potential helpers under the hood some of which are based on
> the mentioned tools that are not part of Emacs proper.  The tools are
> mentioned to encourage the users to install them, so that their Emacs
> environment is more complete and capable.

Let's call them "external tools", then.

> >      +  The commands in this section visit and search all the files listed
> >      +in the @code{xref} backend's database, one by one.  For these
> >      +commands, the database serves only to specify a sequence of files to
> >      +search.
> >
> > That may be true for tags-search, but that's never true for 
> > xref-find-references.
>
> It's true when xref-find-references uses IDUtils, for example.  And
> even when it doesn't, the command does use _some_ database, even if
> that database is made up of doc strings loaded into memory plus
> load-history.

The specific sentence that isn't true is: "For these commands, the 
database serves only to specify a sequence of files to search."

No, not only. It serves for more, IIUC.

Again, here you present unnecessary specifics (backend has a database, 
database is only a list of files), which aren't really true in all cases.

> Once again, look at this from the user POV, not from the POV of
> someone who knows how the implementation works.  Users need some
> mental model of what's going on to make some sense and order out of a
> heap of potentially unrelated features.  The above should provide
> users with a conceptual framework that facilitates coming up with such
> a mental model.  At least I hope it will.

I'd rather the imaginary framework didn't stray too far from reality.

> > If the current backend defines xref-backend-references, then we just ask it, 
> > and show the results.
> >
> > If it doesn't, we delegate to the first available "CEDET tool", but they do 
> > perform the regexp search, not just list the files. Then, xref-collect-references takes 
> > the list of matches and verified, for each one, whether the match begins and ends on a 
> > symbol boundary (Grep can't check that).
>
> I'm not describing xref alone, I'm describing what xref _and_ its
> backend do together.  Differentiating between the two parts of the job
> is not useful for the purposes of the user manual.

You _don't know_ what an arbitrary backend does. Xref just asks it, and 
the backend returns a list. You can describe what elisp and etags 
backends do, but that's not what's happening here.

> In any case, what you describe above is not something that can be
> reasonably said in a user manual, can it?  You describe the (current)
> implementation, whereas the user manual has quite different goals.

No, it's you who seem to be describing the implementation, and end up 
doing it not accurately. I'd be entirely fine with not describing the 
implementation at all.

> You are saying that the description I came up with loses some
> accuracy, which is something I'm aware of, and which is inevitable
> when describing complex software made up of several independent
> modules whose features only partially overlap.

I think we should either describe things accurately, or omit describing 
what we don't know. "Magic happens" is fine, the user just needs to know 
the initial requirements and the end result.

On the other hand, I haven't seem some crucial information mentioned:

- The the databases used by these tools can get out of date, and it's 
the user's responsibility to update them. (Hopefully, we will solve this 
problem sometime).

- Each of the external tools, except for Grep, supports only a fixed set 
of programming languages. If, in a multi-language project, we open a 
file using an exotic-ish language, and ggtags database is present in the 
project root, xref-find-references will return an empty result, erroneously.

> If you have suggestions for how to convey these ideas without going
> into implementation, I'm all ears, of course.

I'd prefer something like:

Xref has a set of commands, which ask for data from the currently active 
Xref backend. And use its identifier-at-point and 
identifier-completion-table logic as well.

And there's this special command called xref-find-references. We wanted 
functionality like that very much, but turns out that both built-in 
backends (elisp and etags), aren't good at this sort of thing.

So we've written a default implementation for xref-backend-references, 
which is used when a backend can't give us better data. That 
implementation uses the "external tools" feature borrowed from CEDET, 
that uses one of the tools from the list, provided the necessary program 
is present on the computer, and a database file has been generated, 
falling back to Grep. We could always use Grep here, but ggtags and 
friends are faster. On the flip side, their databases don't support all 
languages, and can get out of date, we're very sorry about that.

Oh, and by the way, that default implementation also uses the Project 
package, to determine which directories to scan.

> > What list? xref definitely uses one database at a time (per project root, if 
> > we're talking about the default implementation).
>
> Most of the commands described in the section from which this text
> comes are in etags.el, for which the description is accurate.  The
> only other command is M-? (xref-query-replace), for which this is

M-? is xref-find-references.

> somewhat less accurate, but not entirely incorrect: a single database
> is still a special case of "a list of all relevant databases".

This looks confusing to me.

> Once again, feel free to suggest more useful ways of conveying the
> ideas such that they make sense to users.  The above was the best I
> could come up with, and I'll be happy to hear about possible
> improvements, if they exist.

In this case, I'm not entirely sure why that piece of text exists.

> >      address@hidden tags-search
...
> > The last sentence makes it seem like it goes through the xref machinery, 
> > whereas it doesn't.
>
> The last sentence mentions backends because the parent section is
> built upon that concept: that these commands use the appropriate
> backend as they see fit.

And this is blatantly inaccurate in this case. I think that tells us 
that the sections' organization should be different.

> That concept was introduced as an attempt to
> make sense of the fact that xref implemented only part of what's
> available with etags.  I do indeed think that IWBNI xref provided a
> front-end for these features as well

But then, it kind of already does, doesn't it? You can initiate a regexp 
search with project-find-regexp (and even choose the directory with 
C-u), and perform the query-replace from the results buffer.

It would be easy to create dired-specific version of this command, that 
will use the marked directories, if we want to. Just let me know what to 
call it.

> > This makes it seem like tags-loop-continue is applicable for use after 
> > xref-find-* commands as well. But it isn't.
>
> Well, it explicitly references tags-search and comes directly after
> the description of that command.  Would it be better if the first
> sentence said that explicitly, like below?

Probably.

>    Having found one match with @kbd{M-x tags-search}, you probably
>    want to find all the rest.
>
> > Maybe this command, as well as tags-query-replace, the other etags-only 
> > commands, and the related etags variables, should still remain in a separate 
> > section?
>
> No, I don't think so.  It would make two sections out of one, which
> would mean less order in the description.

I really think there should be separation. etags-specific commands, and 
commands that _can_ dispatch to etags (but not necessarily do), and go 
through Xref, are very different beasts. They provide different UI, too.

> Is xref entirely not involved in completion-at-point-functions?

Entirely, yes. But I expect some major and minor modes will 
simultaneously add elements to completion-at-point-functions and 
xref-backend-functions, and those elements could dispatch to the same 
underlying logic under the covers. But the APIs are separate.

> Do we
> still use etags there?

etags by default, elisp-completion-at-point in emacs-lisp-buffers. Other 
major and minor modes set their completion functions as well.

> The original text mentioned etags, and I
> imagined that was replaced by xref now.  (I couldn't trace the code to
> see what was it actually doing.)

You can look at the value of completion-at-point-functions in a c-mode, 
and an emacs-lisp-mode buffers.

> >        mode is enabled, it tries to use the Semantic parser data for
> >        completion (@pxref{Semantic}).  If Semantic mode is not enabled or
> >        fails at performing completion, it tries to complete using the
> >      -selected tags table (@pxref{Tags}).  If in Emacs Lisp mode, it
> >      +available @code{xref} backend (@pxref{Xref}).
> >
> > Also not true. Like mentioned, we don't use xref backends for completion. 
> > Unifying in-buffer completion and xref identifier completion might be 
> > beneficial, but it's not easy to do.
>
> I only changed the cross-reference, since there's no node named "Tags"
> anymore; and etags is one of xref backends.

etags contains an Xref backend (which is mostly a set of functions that 
have both "xref" and "etags" in their names), but I wouldn't call the 
whole etags package an Xref backend.

In particular, tags-completion-at-point-function is not used by the 
Etags Xref backend. They both delegate to tags-lazy-completion-table, 
however.

So I think this sentence confuses things by implying that Xref backends 
are responsible for in-buffer completion. I.e., in practical terms, 
implying that one could register a new Xref backend (by adding an 
element to xref-backend-functions), and that would take care of 
completion support. Which isn't true.

> Are you saying that
> completion-at-point has been changed not to use the tags table when
> the Semantic mode is not enabled?  Or are you saying that this text
> was incorrect even before my changes, and we didn't use tags tables
> even before xref was added?

I wouldn't call the text entirely accurate before (since it only lists 
Emacs Lisp mode as an exception, implying all other modes use etags for 
completion). But it's the new text that I'm really worried about.