Re: Emacs completion matches selection UI

[Dmitry Gutov]

On 06.01.2014 19:51, Toby Cubitt wrote:
> Have you looked at the Completion-UI API for defining a new widget? It's
> uniform across all the widgets. Has to be, otherwise there would be no
> way of allowing it to be extended with third-party widgets (see my
> comment below).

Looking at it now. Seems reasonable, but the data is somewhat coupled to 
the rest of Completion-UI. Examples:

1. `prefix-replaced' and `common-substring' both seem to be tailored to 
the dynamic interface.
2. `completion-source' seems to have to refer to a Completion-UI source, 
right?

So how would one use an UI defined as such, in a different package? 
Would Company define itself as a new source? Or add a source per each 
backend?

> What I meant was that the UIs themselves can act very differently. One
> shouldn't make assumptions that they all behave in a similar way. The API
> itself is uniform.

It's the same in Company. All four base stock frontends are different. 
One displays completions in a dropdown, another shows the (sole) 
completion inline in the buffer, and the other two use the echo area.

> The best comparison one can make is that
> company-backends loosely corresponds to a mixture of
> completion-ui-source-definitions and auto-completion-source-functions.

The latter: yes, the former: no. It's never used the same way as you use 
`completion-ui-source-definitions'.

> > In most places, it's used as a second argument to `assq', so it acts as
> > a "source registry" of sorts, and we don't have a direct correspondence
> > to that in Company.
>
> That's because Company conflates the backend definitions with the
> mechanism for auto-selecting a backend. Completion-UI keeps those
> separate. It makes sense to me to separate these, for two reasons.
> (1) the backend definitions will typically be written by elisp package
> authors; the auto-selection configuration is something users will want to
> customize.

IME, it makes sense in practice because there usually is a set of 
conditions when a backend is applicable, and there's no sense for the 
user to write them out, when the package author can anticipate them.

> (2) I see no reason to limit each source of completions
> (dabbrevs, etags...) to a single choice of when to use it.

The package author can always add one or several new of customization 
variables when the users request it to make the logic more flexible. 
Most of the time, it's usually just the list of major modes the backend 
applies to, if it's major mode-sensitive.

Please take a look at `company-etags' and `company-dabbrev' and see if 
you can point out the situations when the user might find the `prefix' 
code of either too limiting.

> By combining both the choice of whether to use a backend (nil or non-null
> return value) and finding the list of completions (return value) into one
> function, you force all the selection logic to be lumped in together with
> the completion logic. If you want to use the same source of completions
> (dabbrevs, etags...) in multiple different contexts, you either have to
> code *all* the selection logic you'll ever need into the one backend, or
> have multiple backends for the same source of completions.

Or you can define new backends that would do some common checks in 
`prefix' (maybe calling an extracted function with common code) and 
simply delegate all other actions to the respective base backend. 
Implementing this is trivial.

> I find it logically much cleaner to separate the function for completing
> a string using a given source (dabbrevs, etags...), from the function for
> selecting which source of completions to use.

I understand the principle, really. But the more one "cleanly separates" 
code, the harder it can be sometimes to read it, to get the full picture.

> I hate to say this ;-), but maybe `completion-at-point-functions' got
> this right. By having one function (the hook function itself) that deals
> with selecting a completion source, and a separate function (the hook
> function's return value) for actually doing the completion. This both
> keeps the two jobs cleanly separated (as in Completion-UI), *and* keeps
> the two together in the same list.

It makes certain amount of sense, although it looks like it could make 
creating a "merged" completion function more difficult. We'll see.

But again, this separation also makes things more opaque for the user.

> > `auto-completion-source-functions' also could be thought of as similar
> > to `company-backends' because both contain logic of picking a suitable
> > source (the latter by the virtue of backends themselves knowing when
> > they're suitable), but the list of default detection functions (based on
> > text properties, overlays, faces at point) looks more limiting to me.
>
> Why on earth is it limiting?

Note the word "default" above. This specific list is limiting.

But the general approach, while flexible on its surface, complicates 
things if I intend to use any existing sources, written by third 
parties. Because their authors are unlikely to have anticipated the 
logic I'll add in my custom predicate function and to have written any 
code in their packages I might use. Or, at least, that's considerably 
less likely.

Granted, if I only use Completion-UI for its interfaces and never for 
any existing sources, it's not going to be a problem.

> I really don't see how this is any different to writing a new Company
> backend with the particular selection logic you require, and adding it to
> company-backends. Except that you *only* need to code the new selection
> function and add it to the auto-completion-source-functions hook; you
> don't need to duplicate the code for finding the completions if all you
> want to do is use an existing completion source in a new context.

Like mentioned above, delegating the search for completions to an 
existing backend is trivial. These are functions, and as such they are 
stateless. Just call (other-backend 'candidates current-prefix).

> I don't get this at all. How is adding a free-form function to the
> `auto-completion-source-functions' hook any less flexible than adding a
> new backend to company-backends? Except that if you simply want to choose
> the completion source based on a regexp match, face, text-property or
> overlay-property, then you don't even have to write any new code (as you
> would in Company) but can get away with simply configuring a variable.

Problem is, I can never get away with doing just one check. At the very 
least, I need to check both for a regexp (or thing-at-point) match, 
*and* the face (or, better yes, `syntax-ppss' return value). Not to 
mention the major mode, but this check could be conceivably replaced by 
a combination of `add-hook' and a buffer-local `company-backends' value.

I think the only backend we have that does only one check is 
`company-dabbrev'.

> I think you're trying to view Completion-UI through Company
> spectacles. Some Company features don't exist of have limited support in
> Completion-UI (e.g. combining sources), some Completion-UI features don't
> exist or have limited support in Company (e.g. adding new custom UIs),

Adding new UIs is simple, as long as they conform to the 
`company-frontends' interface. I'm not completely sure yet if they're 
decidedly "worse" in some sense than Completion-UI interfaces.

> > You could say that I can add another function or several to
> > `auto-completion-source-functions', but they won't be able to do
> > anything smart with third-party sources, I'll have to take care about
> > each source I might want to use, separately, in those functions.
>
> Why on earth not?! They can do whatever they want - they're functions!
> (OK, anything short of solving the Halting Problem ;-)

"Anything smart" in this context would be to use any metadata the 
authors of third-party sources could have supplied if they were 
encourages to do so.

> Sure, but you have to duplicate the same logic in each backend function,
> and you have to either lump *all* the selection logic you'll ever need
> into one function, or have multiple backends for the same completion
> source.

That depends on what one means by the "same logic". Any logically 
different predicate can be extracted into a separate function. Take a 
look at the existing Company backends, the `prefix' section are usually 
quite succinct.

> Yes, if what you mean is an API for defining completion UI widgets,
> together with a bundle of standard widgets. As far as new widget authors
> are concerned, it would be a completion widget API.
>
> But let's be clear: as far as *completion* package authors are concerned,
> it's not a widget API at all. A completion package won't call the tooltip
> widget, or the popup.el widget, or the dynamic completion widget. It will
> call something like `completion-at-point'.

Actually, I was thinking about the former option. Let's define widgets 
with an API in the usual sense, so it can be used by both 
`completion-at-point' and external packages. We'll need this kind of API 
either way, in order to be able to write new widgets.

> There are really two APIs here:
>
> 1. An API for defining new completion widgets, a customization interface
>     to let the user configure them, and code to hook the new UIs into
>     `completion-at-point', `completion-in-region', and eventually
>     minibuffer completion and the rest.
>
> 2. An API for defining new completion sources and new source selection
>     logic, and a customization interface to let the user configure
>     them. This API already exists (c-a-p-f et al.), and the new UI code
>     will have to integrate with it.

Yes, but see above. Using 1. from Company would be the current next step 
toward integration, as I see it.

> > We also document how to use `company-backends' and, for third-party
> > packages, how to add a specific backend to it. But it's a simple data
> > structure, so for users with some experience just knowing backend
> > function names is sufficient.
>
> Sure, that's equally true of `completion-at-point-functions' and
> `completion-ui-register-source'. (Except that new backends are
> first-class citizens in Completion-UI, and show up automatically as
> options in the completion UI customization interface where
> appropriate. This would be harder to replicate with
> `completion-at-point-functions', but I suspect with enough Customize
> macro trickery it would still be possible with the existing Emacs
> completion backend API.)

Customizing hooks is a tricky business. I believe the opaqueness of 
c-a-p-f to the user is the main problem with the current API.

> > Most Company backends at least have to check that the buffer is in
> > matching major mode.
>
> Whereas in Completion-UI, you just use the standard Emacs mechanisms of
> setting buffer-local variables from you major-mode hook, or using
> `add-hook' with the LOCAL flag set to add a function to the
> auto-completion-source-functions hook.
>
> I prefer to use the standard, tried-and-tested Emacs mechanisms where
> possible (major-mode hooks), rather than creating an entirely new
> package-specific way of configuring things for different major-modes. (I
> don't always stick to this myself, though I probably should ;-)

Emacs core also has custom variables that list major modes applicable 
for a given function. Example: `font-lock-global-modes'.

Imagine you have two sources, each applicable in a different major mode. 
With Company, this means two backend functions, and a modified 
`company-backends' value.

With Completion-UI, with hooks and buffer-local values, just count the 
entities:

- foo completion function
- bar completion function
- foo predicate function
- bar predicate function
- function foo-set-source-functions, setting 
`auto-completion-source-functions' to a relevant local value
- function bar-set-source-functions, doing the same for bar
- (add-hook 'foo-mode-hook 'foo-set-source-functions)
- (add-hook 'bar-mode-hook 'bar-set-source-functions)

...now scale this to 5-10 modes and completion functions and see the 
amount of boilerplate explode.

> So every backend has to look at what's being asked of it every time it's
> invoked, and return nil if it's asked to do something it doesn't support?
> And also check the major-mode and see if it's appropriate every time it's
> invoked, even though the major-mode is very unlikely to change (and runs
> a major-mode hook when it does)?

That's only in the `prefix' call. So, once per user command or so.

> This sounds rather inefficient. I'm sure it's fast enough for Company,
> but I'm skeptical it would be fast enough for predictive-mode's
> auto-completion-mode (where tenths of a second matter).

You just haven't measured it.

  (dotimes (_ 1000) (derived-mode-p 'emacs-lisp-mode))

takes less than 1 ms. IOW, the comparison itself takes less than 1e-6 s.

> It's not that the source returns a keymap or widget-building function
> each time it's called. The keymaps or functions that tweak the UI for a
> specific completion source are configured when the source is registered
> with Completion-UI. The completion function itself just returns
> completions (and associated data).
>
> Hope that makes it clearer.

It does, thanks.

> To summarise:
>
> 2. We're probably stuck with c-a-p-f et al. for the completion source and
>     source-selection API. How would this need to be extended to integrate
>     it with a new Emacs completion UI? How could it be extended to better
>     support Company and others?

To repeat what I said earlier, I'd like to see it customizable by users, 
and I'd like to see a proper merging function.

> 3. The most popular and useful "list-the-available-completions" UI is
>     popup.el. Is there any chance of getting copyright assignment for
>     this? Or will we be forced to code something equivalent from scratch?

https://github.com/auto-complete/popup-el/issues/50