Re: UI inconveniences with M-.

[Eli Zaretskii]

> From: Stefan Monnier <address@hidden>
> Cc: address@hidden,  address@hidden
> Date: Wed, 29 Apr 2015 17:54:04 -0400
> 
> >> In the "M-. find-tag" example, the UI does let you see all matches.
> >> It's just that there's only one.
> > We already agreed that there were at least 2.
> 
> No, we agreed that in some ideal world there could be two.  But the
> backend as it is written now, only returns one for that case.

If the back-end conceals potentially useful information, it should be
fixed not to do so.

> > More importantly, the issue that I was raising in that example is that
> > the results change too radically when the back-end is changed.  IMO,
> > it makes no sense to display more than 140 candidates with one
> > back-end, and only one with another, in the default configuration.
> 
> I agree the 140 "matches" are absurd and need to be fixed.
> The single match of the other backend is what I want, OTOH.

Only when we are 100% sure there is really only one match relevant to
the user.  When we aren't 100% sure, it's better to show a few
additional candidates and rely on the user to disregard them if she
doesn't need them.

IOW, a bit of additional, perhaps redundant information is much less
of a problem than missing information.

That you personally find that information always redundant does not
mean it's true for everyone else.  There's more than one use case for
these queries, see below.

> > Once again, I never said that a single match should be popped in a
> > buffer.  I said that I don't expect to see a single match in this
> > case, because that's a lie!
> 
> No, it's not a lie and I would even argue it's neither a bug nor
> a misfeature.  You expect etags-style information from the xref/elisp
> backend, but that's simply not the info it tries to give you.

Once again, if the back-end conceals potentially useful information,
the back-end needs to be fixed.

While etags shows a very long list of potential matches, it produces
them in the order of importance; I don't remember ever having to
examine more than the first few hits (and thus wasn't even aware of of
the sheer length of the full list).  So with the old UI the
ridiculously long list of candidates was never in my face.  The new
UI, by contrast, makes this acutely evident by showing me the full
list, and on top of that it destroys the order produced by the
back-end, which would allow me to look only at the few first hits.

IOW, the etags back-end by default produces a long list because it
relies on the old front-end to avoid annoying the user by the length
of that list.  The new UI, by contrast, is clearly designed on the
assumption that the list of candidates is very short, so it must
either limit the number of match strategies used by etags, or filter
the results according to some useful criteria.

> > compelled to respond.  But my perspective in filing the bug report was
> > the perspective of a user looking at the results through the UI, and
> > trying to use that UI to manipulate the results.
> 
> See, you use the term "UI" in a completely different way.  When I say
> "UI" in this discussion, I'm talking about the code which takes the info
> from the backend and makes it available to the user.

That's your misunderstanding.  I described my user experience from
using this new feature; I never said where the fixes should be made,
primarily because I didn't understand the code well enough (and still
don't) to say anything intelligent about it.

If I wanted to talk about the code, I'd say something like
"this-and-that function does wrong things because of so-and-so".

> So the fact that one backend returns 1 item and the other returns 140 of
> them is completely outside of the control of the UI.

Says you.  Through my naive-user eyes, filtering 140 hits to provide
just a few is perfectly within the capabilities of the UI, at least in
principle.

> Maybe the resulting end-user experience sucks, but that's not the fault
> of the UI but the fault of the backends.

It sounds more and more like it could be the fault of the design, and
specifically of how functionality is divided between the back-end and
the UI.  Let me elaborate.

>From the rest of the discussion, and the related threads spawned by
it, I seem to get the feeling that the design was explicitly meant to
leave the UI completely oblivious to the subject matter, and instead
infinitely trust the back-end to produce "the right" results.  If so,
I think this is a design mistake, and this discussion (and my
frustrating user experience) is the result.  IME, an interactive
user-level feature that supports multiple back-end engines cannot
avoid some degree of subject-matter knowledge and some kind of
manipulation of the results that come from the back-end, or else it
won't be able to provide consistent user experience across the
supported back-ends.  Having such a "dumb" UI in this case will cause
complications, some of which we already see:

  . the need to "fix" the back-ends, and continue "fixing" them in the
    future
  . the need to merge back-ends and invent some suitable
    infrastructure for that
  . etc.

To me, this means that separating the back-ends from the UI while
leaving the UI "dumb" is basically unworkable, because such a
separation does not really separate anything: there will still be a
very high degree of coherency and cohesion between the two parts.  Or
maybe there will only be one usable back-end, and the rest will
bit-rot.

IOW, the separation of functionality is in the wrong place.  To be
useful, some of the "smarts" need to be on the UI side, where user
control can be best implemented, and where user intent is known much
better.  Then we could stop caring about back-ends providing too much
info, because the filtering will be in the UI and, most importantly,
by uniform criteria.  And even if the filtering is in some of the
back-ends, it will be according to UI's guidance, whereby the UI will
convert the user-level control options into the language
understandable by the back-end.  That is the only practical way of
providing a consistent user experience with multiple back-ends.

> >> >> Arguably, the find-tag advice in org-ctags.el could be offered as well,
> >> >> if org-ctags.el happens to be loaded
> >> > ??? Why should the xref matches depend on what is and isn't loaded?
> >> Because that's how xref/elisp works.
> > If so, then it's much less useful than I thought.
> 
> And to me, it makes it more useful.
> 
> > I don't see how such a back-end can even be a candidate for becoming
> > the default.
> 
> And I don't see how we've live with the etags.el UI for so many years.

How frequently did you need to study a large body of code written by
someone else, like a large project where you need to fix a bug?

I do that a lot, both on my daytime job, and in Free Software related
moonlighting.  If you seldom, or never, have to quickly find your way
in such vast unchartered territories, and mainly work with code that
either you yourself wrote or with which you are intimately familiar,
then I can understand why you don't like more than one match
candidate.

Likewise, people who mainly work with small projects, where the code
line count is in hundreds or a small number of thousands, might never
need the less-than-exact matches provided by etags: Grep is good
enough and fast enough for that.  But you cannot do without etags in a
larger project.  (You also cannot do without being able to find all
references to an identifier, which is why I'm using ID Utils for the
past 15 years, and was disappointed to learn that xref-find-references
has no back-ends that implemented this kind of search.)

I very much hope Emacs will continue to be able to support the kind of
activities I described above, which AFAIK are very important part of a
software developer's job throughout the industry.

> I've tried several times to make real use of it, but always found it
> completely unpalatable.  What with having to build those damn TAGS
> files, remember to refresh them, remember where they are, constantly
> tell Emacs again where they are, deal with its inability to find the
> right spot and having to repeat the "C-u M-." finger gymnastics
> umpteen times.

Those are exaggerations.  Building TAGS is almost instant, even in
Emacs, you need only refresh them very seldom, and Emacs offers the
place from which to load them so you don't need to remember.

But this, again, is immaterial for this discussion.  I hope you will
agree that, whatever issues we have with etags, replacing it with
something that lacks important functionality is not a good idea.

> xref/elisp may not let you jump to those functions/vars that aren't yet
> loaded, but beside this little detail, it *just works* without any
> extra setup, nothing.

That "little detail" all but invalidates most of my use cases.  We
seem to have very different development experiences, probably due to
what each one of us does most of the time, but I hope you can see that
there are valid use cases outside of your personal experience, and
Emacs should support them if it wants to be a good IDE.

And I'm not even sure your ideas of how to solve that "little detail"
are workable, because of the potentially adverse consequences of
loading code you don't actually need (or want) to execute.  What if
the code is buggy, or dangerous, or simply does things you don't want
to be done in your Emacs session?

And how do you scale this up to packages that are outside of
load-path, for whatever reasons?  Etc. etc.

> >> Not at all.  Many (most?) packages which offer a functionality along the
> >> lines of "M-. to jump to the definition" use an implementation technique
> >> which is fundamentally similar/equivalent (obviously, they don't query
> >> a running Emacs, but they query a running REPL).
> > Think about compiled languages.
> 
> AFAIK, JDEE used the same approach for Java.

I don't know what that means, and don't know enough about JDEE to talk
about it.  In any case, Java is not a "classic" compiled language, as
a Jave development environment is generally capable of running the
code in an interpreter.

> I don't claim that all backends will have to work this way.  Just that
> it's a common implementation technique.

Common for some paradigms of development activity, for some
development tasks, and probably also for some programming languages.
It's not universally true by any measure.

> > Until that problem is solved in core, I don't see how a back-end that
> > requires to load a package can be the default.
> 
> Agreed.

But since it was already made the default, it means we must fix this
ASAP.

> >> > That's one way of looking at it.  Another is to say that etags gave
> >> > you more information and thus allowed to find more loose matches,
> >> > which is helpful when your memory is failing you.
> >> M-. jumps to the definition of the "thing under point".  There's no
> >> memory involved.
> > I was talking about "C-u M-."
> 
> C-u M-. also lets you do loose matching, via completion, if your memory
> is failing you.

I don't think completion is the right tool for these searches, because
the name alone doesn't tell enough.  So if we want to base that on
name-completion, we are asking the user to type "something TAB" every
time she needs another candidate.  I don't think this is a very
convenient UI for this job.

> > I didn't say "unfiltered".  I asked for user control of the amount of
> > filtering or of looseness of the matches.  In etags.el terms, think
> > about giving the user control on what goes into find-tag-tag-order.
> 
> That's OK, then.  It seems that etags is simply unable to give
> sufficiently reliable data to filter it effectively, so we're stuck with
> heuristics and user-knobs to choose where they're rather get bitten.

No.  etags provided you a long list, but ordered by importance.  As I
said, I don't remember ever having to examine more than the few first
candidates.  etags relied on this order to save the user from having
to deal with a huge list of potential candidates, at least in most
cases.  etags _can_ effectively filter the results it produces, if you
ask it to use less members in the find-tag-tag-order list.  Or you
could just take the first N candidates of the list it returns, and
ignore the rest.

> > Once again, I question the decision to switch to this UI, when we
> 
> I know, but that's just a waste of time.  The backends won't come before
> this system is standard and common place (and they'll take their time,
> because in the mean time they can keep using their ad-hoc UI which also
> works on older Emacsen).

See above: you assume that the division of functionality between the
UI and the back-ends is at the right place.  I'm not so sure.  If I'm
right, then when more back-ends come, we will see more problems, not
less.

> > It's not!!  There are partial matches shown by completion -- they are
> > the next candidates.
> 
> Again, you choose to look at it this way because you're used to the way
> etags.el worked.

Not to how etags.el worked, to how _I_ worked, what _I_ needed to do
with etags.el's help.  We simply have 2 very different kinds of jobs
to do most of our time.  One more argument for more user control on
the UI level, if you ask me.

> But to me, "find-tag" and "find-tag-tag" are not two different
> matches to my request.  They're just two completely unrelated
> things: either I'm looking for one or I'm looking for the other.

Assuming you know what you are looking for, yes.  I described a
situation that is frequent for me where you generally don't, at least
not well enough to be satisfied by a single exact match.

> The design of xref is based on the hope that the backend can return
> a very small list of candidates, ideally of length 1.

And therein lies its weakness.  I actually don't understand how this
kind of assumption could be allowed to exist, when the _default_
back-end, and one of only 3 existing ones, blows that assumption out
of the water.

> On the contrary, I want a tool that's precise, and gets me directly to
> the only corresponding definition (in the 99% of the cases where
> there's indeed only one) with a single key-press and no questions asked.

Yes, I want that too.  I also want a tool which will write code for
me.  I just don't believe either of these is possible, because in most
of my use cases (which I believe are shared by many in the industry)
no program can ever be lucky enough to guess what definition is that.