Re: Emacs completion matches selection UI

[Dmitry Gutov]

On 08.01.2014 06:58, Toby Cubitt wrote:
> > I don't really see why a non-prefix completion UI (or any other one) has
> > to modify the buffer text. Company has an inline "preview" frontend, but
> > it uses an overlay, and so the buffer text remains unmodified.
>
> Ah, you mean using the `before-string' or `after-string' overlay
> property? That does sound like a better implementation.

Yep. Company's popup and popup.el are also implemented using this property.

> Neither Company nor CAPF support refining the completion list by adding
> more characters to the prefix/pattern (somewhat similar to
> isearch). Completion-UI does, and it's very useful. Indeed, this is one
> of the desired features Ted explicitly asked for in his recent post.

Company has as isearch-like function (press `C-s' when you see the 
completions popup). It doesn't *add* to the pattern as much as just 
searches within the candidates list. It could refine it, though.
Again, there's no buffer modification going on during that.

> At the very least, you need to store which completion backend
> was used (which I think Company does too; CAPF makes this trickier, since
> it doesn't have a simple way of identifying completion sources).

You may be able to get away with just storing the value returned by the 
completion function. Instead of refreshing the completion boundaries 
after each command, you could just assume that they stay in place, and 
the closing boundary just moves according to the amount of text 
inserted/removed during completion.

> If the former, we'll very likely want to record which CAPF source
> returned the completions somehow. If the latter, then the equivalent of
> the backend name is fine. (In the current Completion-UI implementation,
> this property stores the completion source name, which modulo
> implementation differences is directly equivalent to your backend names.)

Come to think of it, that should be easy to do in 
`completion--capf-wrapper', as long as FUN is a symbol, not an anonymous 
function.

> But how would we use that in the Customization interface, to allow users
> to customize the UI in different ways for different sources?

Hm, maybe the function name too, see above. We'll need to retain all 
PROPS anyway.

> What I don't see at the
> moment is how Customize would discover what all the possible CAPF source
> names are, in order to conveniently list them as options in a menu.

I don't really see that happening at all.

> > The opaqueness in c-a-p-f is bad because the exact values of
> > `company-backends' and `completion-at-point-functions' are significant.
> > A third-party package author can only push a function in either of these
> > lists, but they can be responsible for the eventual order of the elements.
> >
> > And the order matters, because it influences which completion backend
> > will get picked when several are suitable. When we get to the grouped
> > backends, which I've mentioned several times, which Company supports,
> > and CAPF will hopefully support in the future, being able to understand
> > and maybe change the values of either list becomes even more important.
>
> I completely agree.
>
> How do you suggest we could improve it, without replacing CAPF or
> breaking backwards compatibility?
>
> This is something you care about more and have more experience of in
> Company than I do with Predictive. I'm pretty confident that I can adapt
> the Completion-UI code to whatever API we settle on for defining
> completion sources and selection logic, without too much pain.

Sorry, no idea. `company-backends' is not a hook, and this simple nature 
makes it more transparent, while somewhat less powerful. Making c-a-p-f 
less powerful is probably a non-starter.

> > My question was, if you could point out problems with any of our bundled
> > backends (or, failing that, third-party ones). If they look okay, maybe
>
> Did some text get cut off here in your reply?

Sorry, forgot to finish it. Roughly the idea is this: if the backends we 
bundle look okay and flexible enough, maybe the API is sounds for most 
cases, barring extreme examples.

Anyway, this doesn't matter much, because we'll be refining CAPF.

> > 1) Adding some pre-completion hook which would allow you to run some
> > code once, set a buffer-local variable, which all backend functions for
> > LaTeX can refer to later in `prefix' action.
>
> I don't think this can work, because the choice of source depends on the
> location of point and the text in the buffer, which changes from
> completion to completion.

One *per completion*, of course. As opposed to once per backend per 
completion.

> For the Predictive LaTeX support, I use a mish mash of piggy-backing on
> jit-lock where possible, some regexp match tests, and a limited form of
> incremental parsing of the LaTeX code when the rest aren't enough. (I
> used to exclusively use the latter, implemented using my auto-overlays
> package. Surprisingly this was more than efficient enough to cause no
> noticeable delay when typing and auto-completing, even on very old
> hardware. But writing and debugging the incremental parser definitions
> were a nightmare, largely because TeX doesn't have a well-defined
> grammar. So I replaced as much as possible of the parsing with simpler
> methods.)

Sounds scary.

> This effectively means moving the selection logic out of Company and into
> Predictive. Which isn't necessarily a bad solution. It's how I used to do
> it in Predictive, until I generalised the selection logic and moved it
> into Completion-UI so other sources could make use of it.
>
> If we stick with the CAPF API, I suspect I'll end up moving my source
> selection logic back into Predictive, and making it Predictive-specific
> again.

I don't think you have to. A completion function can similarly return 
different completion tables based on the context.

> > Regexps we have already (company-grab-...),
>
> How fast is this if you have to go through, say, 100 moderately complex
> regexps checking for a match? (See the predictive-latex.el in the old
> Predictive tarball release for examples.)

I'm not really sure what to look at. `predictive-latex-load-regexps'? I 
probably won't be able to measure it without rewriting that code.

> In Completion-UI, I implemented combined completion sources through a
> `completion-ui-combining-complete' wrapper function. (They're not
> functionally equivalent to your merged sources, but the requirements are
> somewhat similar.)
>
> Would using something like that to build merged CAPF functions be a
> solution for Company? I know this isn't particularly user-friendly if you
> want to allow users to easily define their own merged sources. I don't
> know if that's something people regularly do in Company.

In Company, you just add a list value to `company-backends'. There is 
one in the default value already. Its handling is not perfect, but does 
the job currently, as long as the backends are compatible enough.

> There's one thing we should perhaps think a bit more about.
>
> Is it right to say that the majority of the Company backends are selected
> based on global properties of a buffer (e.g. major-modes)? This seems to
> be the case for the default `company-backends' list. In Completion-UI I
> was almost entirely focused on selecting backends based on local
> properties at different locations within a buffer (regexps, faces,
> syntax, etc.).
>
> How do you envisage supporting local source selection in the new (or
> enhanced CAPF) API? Would this kind of local completion source selection
> always be implemented within a single backend, like the Company CEDET
> backend? Or should there be a convenient way of supporting it in the
> generic source selection API?

Generally, you'd just replace several existing entries in 
`completion-at-point-functions' with a new one that would first check 
local properties and then call an appropriate function among the 
replaced ones. Or, if the new function does not handle existing 
completion functions, scratch out he "replace[d]" bit.

> Following the code path is unnecessarily convoluted. I can't tell that
> company-dabbrev-code internally delegates to the company-dabbrev backend
> just from looking at `company-backends'. If it looked more like:
>
> ((use-dabbrev-code-p . dabbrev-completion)
>   (use-dabbrev-p . dabbrev-completion))
>
> it would be immediately obvious that these are two different ways of
> using the same source of completions.

This wouldn't work because company-dabbrev-code uses case-sensitive 
search and doesn't downcase the results, as opposed to company-dabbrev. 
But they share code anyway.

> The question for now is, what should go in the generic Emacs API? Vanilla
> CAPF, perhaps with some additional standard PROPS properties? If the
> generic Emacs API can't replace company-backends in Company, then in my
> view the API is no good.
>
> You're the expert here :)

So far I don't see obvious missing parts in CAPF API. But there are some 
that could be harder to support in frontends, like partial completion, 
and it also seems to be suboptimal for implementing backends with 
external services, see the tangential discussion at 
http://debbugs.gnu.org/cgi/bugreport.cgi?bug=16334.

> The way I imagine it, completion packages like Company would rarely need
> to use the API for defining new widgets. (Unless they're defining custom
> widgets that are so package-specific they don't belong in core Emacs. But
> that seems like a rare case.)
>
> What they *will* all use is the API for *invoking* the completion
> interface, which we began to discuss below.

Yep.

> Erm, I don't see any major shortcomings in the API. Probably that hasn't
> come across in the discussion :-)
>
> Minor ones that have come here were:
>
> - No support for non-prefix completion (but it looks easy enough to
>    extend the API).

So far I don't see an easy way to visualize non-prefix completion in 
frontends. Specifically, the way candidates relate to an input string. 
With prefix completion, it's easy: just highlight the prefix, but 
partial completion, for example, requires re-parsing the returned 
candidates on the fly, to try to match the separate letters.

> - Difficult (impossible?) to distinguish between "no completions
>    available" and "backend not applicable".

That's easy. "Backend not applicable" means `prefix' returned nil. "No 
completion available" means `prefix' returned non-nil, but `candidates' 
returned nil.

> OK, makes sense. If I understand correctly, CAPF is fine for letting
> users decide the order of trying backends.

I'm not sure. When someone's using `add-hook', you don't really look at 
the order of all elements, you just add a new one at the beginning or 
the end of the list. Should be all right most of the time, though.

> It would be nice to define a Customization interface that lists the
> available CAPF functions in a drop-down menu (whilst also allowing
> free-form entries, of course). If I find a way of porting the
> Completion-UI auto-updating customization interface magic to CAPF, this
> would be feasible...

This looks impossible to me. CAPF functions are defined in many places, 
including third-party packages, using plain `defun' interface. Can there 
really be a way to track them?