Re: Emacs completion matches selection UI

[Dmitry Gutov]

On 06.01.2014 06:39, Toby Cubitt wrote:
> I'm sure if you'd built Company on top of Completion-UI, we would have
> rapidly improved the APIs to support everything you needed as Company
> grew.

I'm just the maintainer. :) Company's author is Nikolaj Schumacher, 
though he's stepped away from its development.

> I originally thought I'd want to disable the learning algorithms once I'd
> trained Predictive on a corpus of my writing, for precisely the reason
> you describe. But in practice the order is stable enough that I never
> found it necessary.

I'll have to look into it.

> Completion-UI stores state by marking the location of the in-progress
> completion with an overlay, and storing data in overlay properties. You
> could alternatively use text properties, as you mention later. But I find
> overlays a better match for this kind of thing.

I mentioned text properties in the context of merged backends, so the 
properties would be on the candidate strings, not in the buffer. That's 
not interchangeable with using an overlay.

> (Presumably Company does something similar.)

It does store some data, as an implementation detail, but in no way that 
data is sufficient for a front-end to handle a sequence of commands on 
its own. Like I said, the front-ends use the buffer-local vars. Multiple 
pending completions have never been a goal so far.

> > I'd still like to a natural way to extend such widgets with new
> > commands. Passing a keymap as argument, or rebinding a dynamic variable
> > that points to keymap works, I guess, but it's not very nice.
>
> Well, this is only meaningful for some of the widgets. Indeed, "widgets"
> is mislead; the completion UIs in Completion-UI aren't necessarily
> widgets at all. For example, there are no "commands" at all in the
> dynamic completion UI, nor is it a "widget" in any normal sense of the
> word.

I'd call "widgets" any UIs that encapsulate data (store it in the 
associated overlay, for example) and define interaction commands (e.g. 
have an overlay keymap).

> The UIs absolutely cannot be treated uniformly, because not all of them
> are "display a list of completions"-style UIs. E.g. the "dynamic
> completion" UI is a completely different beast.

That might be unfortunate from the API standpoint, making it more complex.

> Even amongst the "display a list of completions"-style UIs, the
> mouse-driven toolkit menus cannot be used in the terminal, nor can pop-up
> frames, nor can tooltips. That doesn't mean one should not provide these
> interfaces at all as options. It just means they probably aren't good
> default options.

The problem of "cannot be used in the terminal", at least, can be solved 
by doing a check at runtime once, and picking between a graphical and an 
overlay-based interface. As long as they provide similar interaction, 
I'd call that "treated uniformly".

> > Not really. `company-backends' corresponds directly to
> > `auto-completion-source',
>
> Not quite. `company-backends' corresponds to the
> `completion-ui-source-definitions' variable.

I'm inclined to stand by my previous assessment: nowhere in the code I 
see you iterate over `completion-ui-source-definitions' at runtime. In 
most places, it's used as a second argument to `assq', so it acts as a 
"source registry" of sorts, and we don't have a direct correspondence to 
that in Company.

The only place it's iterated over is in 
`completion-ui-customize-list-sources', which provides the list of 
possible values to customize `auto-completion-default-source' (in the 
Git version; I've been looking at the contents of the tarball previously).

> > only instead of one function we have a list of
> > them (which can also be nested one level deep, but that's an extra feature).
>
> Sure, Completion-UI also allows a list of completion sources and selects
> between them automagically. It even supports the same "extra feature" of
> nesting sources one level deep (Combined source, in Completion-UI
> terminology).

I couldn't find the word "combined" in completion-ui.el. Could you point 
at where it's used?

> But the API for this isn't auto-completion-default-source(*). That merely
> sets the default fall-back. The API for auto-selecting from a list of
> sources is provided by `auto-completion-source-regexps',
> `auto-completion-source-faces' and the `auto-completion-source-functions'
> hook. (The former two are in fact variables used by functions in the
> default `auto-completion-source-functions' definition.)

`auto-completion-source-functions' also could be thought of as similar 
to `company-backends' because both contain logic of picking a suitable 
source (the latter by the virtue of backends themselves knowing when 
they're suitable), but the list of default detection functions (based on 
text properties, overlays, faces at point) looks more limiting to me.

Granted, the first two source functions serve to support the explicit 
overriding of the used source (feature not presend in Company, but I've 
never seen a request for this so far), but the second two offer only 
choices based on regexp or face at point. Which is not terrible, but 
more limiting than Company's free-form expectation that 
(backend-function 'prefix) returns nil when it's not applicable at point 
in the current buffer.

You could say that I can add another function or several to 
`auto-completion-source-functions', but they won't be able to do 
anything smart with third-party sources, I'll have to take care about 
each source I might want to use, separately, in those functions.

> (*) I'm referring here to the git version of Completion-UI; the last
> tarball release - which is woefully outdated - used a slightly different
> API.

Thanks for mentioning this.

> Yup, Completion-UI does something very similar. Except that completion
> source (backend) selection isn't directly tied to the backend. It's
> configured independently. So if you need to, you can set up different
> conditions under which a particular backend will be used in different
> buffers, major-modes, minor-modes, or whatever.

Note that there's nothing stopping you from defining multiple conditions 
for when a particular Company backend can be used. Most do that already: 
the popular predicates are "does the major mode match this list", "is a 
particular package loaded", "is the parse state at point in-string or 
outside", "is this buffer visiting a file", "is a particular program 
present in the system", "is there a file with a given name in a parent 
directory" and "is point inside a symbol, after a symbol or outside of 
any symbols".

> Anyhow, I don't think company-mode is something that necessarily belongs
> in a generic Emacs completion UI. At most, Emacs should include the
> necessary definitions to hook the basic completion methods that come with
> Emacs into the new generic UI (dabbrev, etags, minibuffer completion,
> maybe a few others like elisp symbol completion...).

Maybe so.

> Indeed, I would have argued that we should *first* come up with a generic
> Emacs completion UI and API *without* including any mechanism for
> automatically selecting between completion sources.

In other words, a programmatic widget API. This has come up already in 
the discussion with Ted.

> Registering a source just (1) makes life more convenient: you set all the
> option values once when you define the completion source (backend), and
> then you can refer to that source and all its associated options using a
> single name (symbol). And (2) it makes the source available as an option
> in the Customization interface, as you noted.

I would think that, since Completion-UI is less user-facing and more 
about writing Elisp to wire sources and UIs, the Customize interface is 
considerably less important.

> > It makes matters a bit worse for third-party backends, but not by much,
> > I think.
>
> I agree, this is a minor difference, and Company doesn't lose much by it,
> especially since it bundles almost all the backend definitions you're
> ever likely to want.

We also document how to use `company-backends' and, for third-party 
packages, how to add a specific backend to it. But it's a simple data 
structure, so for users with some experience just knowing backend 
function names is sufficient.

> The optional macro arguments only come into play if you're doing
> something more sophisticated (like extending the default completion
> widgets, or overriding the default method of determining the text around
> point that's to be completed, etc.)

Most Company backends at least have to check that the buffer is in 
matching major mode.

> > This pattern is used in many places in Emacs core, too.
>
> In principle, this sounds like a perfectly reasonable API choice for
> simple settings.
>
> Though I don't completely get how it works here. What if a particular
> completion function can't answer some types of question?

Then we use the default setting, behavior, etc. It is somewhat limiting 
(we can't discern whether a backend does not know about a question or it 
consciously returns nil), but so far it's worked well nevertheless.

> E.g. maybe it
> can expand prefixes, but it can't find regexp matches. What does the
> completion function do if it's then asked for a regexp match? Or does the
> backend answer "I can't answer this type of question" and Company moves
> onto the next backend in the list?

We don't have support for regexp matches, but "can't answer" + "move on" 
is a reasonable mechanism to support them in select backends.

Or, to approach it another way, it may be a backend's choice to 
interpret the prefix (or, more generally, the completion string, if it's 
allowed to span after the point) as a regexp. For example, we have this 
as a feature request now: 
https://github.com/company-mode/company-mode/issues/45#issuecomment-31564029

Aside from handling the prefix->candidate replacement more carefully, 
the only thing we'll have to worry about, I believe, is that the 
front-ends won't be so sure that candidate starts with "prefix". For 
`company-pseudo-tooltip', that means no highlighting of the "common" 
part (or using a more complex algorithm), and it could mean different 
things for `company-preview-if-just-one-frontend'. For now, I'm inclined 
to just refrain from using it if the sole candidate does not start with 
"prefix".

> However, this API is not so convenient if you want to pass more complex
> settings, such as extending the UI for a particular source. You'd either
> have to pass the same list of hook functions, keymaps etc. every time you
> call the function that invokes the completion UI for the source. Or you'd
> have to let-bind the same set of variables every time. Which would be a
> pain.

Sorry, I don't understand. Why would you extend a UI for some source? 
Sources provide data, how can they have associated keymaps?

> This pattern is used in many places in Emacs core, too: `defcustom',
> `defstruct', `defun*'... (Perhaps I should have called the macro
> `defcompletion' instead of `completion-ui-register-source' :)

Sure, Lisps have a long history of using and promoting macros, but 
there's also a general recommendation to not write a new macro unless 
you'll really really benefit from it. Maybe you do.

> Also, the "pass the type of completion as the first argument" API means
> you have a hard-coded set of "types" of completion. As usual, since it
> was intended to be a generic completion Elisp library and not a
> fully-fledged user-level completion framework, Completion-UI doesn't
> impose any such restriction, but tries to be as flexible and general as
> possible. I'm not saying this is necessarily a good thing; it could well
> be that this flexibility is overkill.

See above. We only have prefix completion for now, so the first argument 
definitely doesn't choose the type. See the list of possible values in 
the `company-backends' docstring.

> Which bodes well for coming up with a generic Emacs completion UI API and
> code. If we've converged on a broadly similar set of features and APIs
> for the UI parts of Completion-UI and Company, there's a reasonable
> chance they're the features people will want from a generic Emacs API.

Yes, probably.