Re: Emacs completion matches selection UI

[Bob Bobeck]

This has got to be the longest stream of crap I've ever seen pushed
out on this mailing list. Are you all a bunch of novelists or are you
hackers? Pinch this turd off and start coding or shut the hell up.

On 1/7/14, Toby Cubitt <address@hidden> wrote:
> On Wed, Jan 08, 2014 at 03:23:21AM +0400, Dmitry Gutov wrote:
>> On 07.01.2014 07:32, Toby Cubitt wrote:
>> >> Examples:
>> >> 1. `prefix-replaced' and `common-substring' both seem to be tailored
>> >> to
>> >> the dynamic interface.
>> >
>> > Not at all. `prefix-replaced' tells you that the buffer substring
>> > that is being completed has already been replaced with new text in
>> > the buffer. This is important for non-prefix completion (e.g. pattern
>> > matches).
>>
>> > I'm not sure if it's generally applicable.
> [...]
>> > Any UI that modifies the buffer text will almost certainly need to set
>> > `prefix-replaced';
>>
>> I don't really see why a non-prefix completion UI (or any other one) has
>> to modify the buffer text. Company has an inline "preview" frontend, but
>> it uses an overlay, and so the buffer text remains unmodified.
>
> Ah, you mean using the `before-string' or `after-string' overlay
> property? That does sound like a better implementation.
>
>
>> AFAIK, neither Company nor CAPF keep this kind of history: if the
>> buffer was modified, what happens after is a new completion, or no
>> completion at all (if circumstances told us we should abort).
>
> Neither Company nor CAPF support refining the completion list by adding
> more characters to the prefix/pattern (somewhat similar to
> isearch). Completion-UI does, and it's very useful. Indeed, this is one
> of the desired features Ted explicitly asked for in his recent post.
>
> In general, this may require storing a bit more history about the
> completion process, so you can continue from where you left off
> correctly. At the very least, you need to store which completion backend
> was used (which I think Company does too; CAPF makes this trickier, since
> it doesn't have a simple way of identifying completion sources).
>
> You've convinced me that prefix-replaced isn't needed, as long as UIs
> aren't allowed to manipulate the buffer text. I need to think about it
> more, but perhaps that could be a good restriction to impose as it
> prevents UI bugs from mangling the buffer text.
>
>
>> > And the core Completion-UI code that ties all the UIs together makes
>> > heavy use of the `prefix-replaced' property. So even in Completion-UI
>> > it's not only used by the dynamic interface.
>>
>> Could you explain how?
>
> I didn't impose any restriction on the UIs in what they're allowed to do
> with the buffer text, so the core uses this e.g. when figuring out what
> text it should leave behind in the buffer when a completion is accepted.
>
> But I think imposing a restriction that UIs don't modify the buffer text
> during the completion process is probably a good one, in which case it
> may be unnecessary.
>
>> > Similarly, `common-substring' demarcates the longest common substring
>> > if
>> > completion text has been inserted in the buffer, which is information a
>> > third-party UI might very well want to make use of.
>>
>> Why? How?
>
> Who knows how someone might want to make use of it in their UI? They
> might want to highlight the common substring differently, say. But this
> one's just for convenience anyway. A UI can recalculate it from the
> completion list in any case.
>
>> > The properties documented in the `completion-ui-register-interface'
>> > docstring are those that have a standard meaning in Completion-UI. Note
>> > that nothing prevents a particular UI widget from storing its own data
>> > in
>> > its own widget-specific overlay properties if so desired.
>>
>> Sure.
>>
>> >> 2. `completion-source' seems to have to refer to a Completion-UI
>> >> source, right?
>> >>
>> >> So how would one use an UI defined as such, in a different package?
>> >> Would Company define itself as a new source? Or add a source per each
>> >> backend?
>> >
>> > Depends on what the generic Emacs API for specifying completion
>> > sources ends up looking like.  If we end up using c-a-p-f,
>> > `completion-source' will need to store something that identifies the
>> > c-a-p-f entry that returned the completion candidates for this
>> > completion process.
>>
>> Hmm, I was expecting something easier, since IIRC you said Company
>> should have been able to use it without major changes.
>
> Maybe I misunderstood what you were asking. What I meant was, until we've
> settled on the API for completion sources, we can't decide on the format
> of the data that records which completion source ended up being used.
>
> If you meant what would Company have done in the *current* Completion-UI
> implementation, then yes: Company would have simply defined its backends
> via the Completion-UI API, instead of via CAPF or company-backends.
>
> But we seem to heading towards a different API for defining and selecting
> completion sources. Probably CAPF, but perhaps something more similar to
> company-backends.
>
> If the former, we'll very likely want to record which CAPF source
> returned the completions somehow. If the latter, then the equivalent of
> the backend name is fine. (In the current Completion-UI implementation,
> this property stores the completion source name, which modulo
> implementation differences is directly equivalent to your backend names.)
>
> I think you're reading way too much into the details of the current
> Completion-UI implementation. The details will inevitably change if we
> integrate it into part of a generic Emacs completion API. But whatever
> the specifics of the data stored, and whatever we call the
> variable/property that stores that data, we'll still need some way of
> recording which completion source returned the completions we're dealing
> with. Without this information, a number of features become tricky or
> impossible to implement in the UI. As you surely know, since both Company
> already records and uses information about which backend was used, just
> like Completion-UI does.
>
>> > If we end up using c-a-p-f, `completion-source'
>> > will need to store something that identifies the c-a-p-f entry that
>> > returned the completion candidates for this completion process.
>>
>> Probably just the value returned by the successful completion function:
>> start, end, collection (aka completion table, which is often a function).
>
> But how would we use that in the Customization interface, to allow users
> to customize the UI in different ways for different sources?
>
> It seems useful to have this feature, so users can make use it if they
> want to. (See the use-case examples I gave in my reply to Stefan.) I'd be
> sorry to lose the per-source customization feature. It's one I make light
> but frequent use of myself. And users are bound to ask for something like
> this later!
>
> It's a bummer that CAPF doesn't associate unique names with completion
> sources, in the way that Company and Completion-UI do.
>
> One possibility might be to have CAPF functions optionally set a :name
> property in the optional PROPS return value if they want to support
> per-source UI customization for that source. What I don't see at the
> moment is how Customize would discover what all the possible CAPF source
> names are, in order to conveniently list them as options in a menu. (In
> Completion-UI, this is one of the things handled by the macro used to
> define new sources.) But that's a detail.
>
>
>> >>> The best comparison one can make is that
>> >>> company-backends loosely corresponds to a mixture of
>> >>> completion-ui-source-definitions and
>> >>> auto-completion-source-functions.
>> >>
>> >> The latter: yes, the former: no. It's never used the same way as you
>> >> use
>> >> `completion-ui-source-definitions'.
>> >
>> > I said "loosely corresponds", and I stand by that. One of the things a
>> > `company-backends' entry does is to define when a given completion
>> > source
>> > should be used. In Completion-UI, that part of company-backends' role
>> > is
>> > performed by auto-completion-source-functions.
>>
>> Yes, that's why I agreed with this comparison (and also made it earlier,
>> IIRC). I disagree with the other comparison.
>
> Sorry, my bad. I read what you wrote the wrong way around.
>
>> > Could well be. But you seem to be arguing both ways. You also argue
>> > that
>> > the c-a-p-f API is bad because it's opaque and hard for users to
>> > configure. If backend definitions and selection logic are always
>> > supplied
>> > by package authors, then the fact that c-a-p-f is opaque isn't so
>> > significant.
>>
>> The opaqueness in c-a-p-f is bad because the exact values of
>> `company-backends' and `completion-at-point-functions' are significant.
>> A third-party package author can only push a function in either of these
>> lists, but they can be responsible for the eventual order of the
>> elements.
>>
>> And the order matters, because it influences which completion backend
>> will get picked when several are suitable. When we get to the grouped
>> backends, which I've mentioned several times, which Company supports,
>> and CAPF will hopefully support in the future, being able to understand
>> and maybe change the values of either list becomes even more important.
>
> I completely agree.
>
> How do you suggest we could improve it, without replacing CAPF or
> breaking backwards compatibility?
>
> This is something you care about more and have more experience of in
> Company than I do with Predictive. I'm pretty confident that I can adapt
> the Completion-UI code to whatever API we settle on for defining
> completion sources and selection logic, without too much pain.
>
>> >> Please take a look at `company-etags' and `company-dabbrev' and see if
>> >> you can point out the situations when the user might find the `prefix'
>> >> code of either too limiting.
>> >
>> > dabbrev and etags are both sources that, if they're useful in a given
>> > buffer, they're useful in the entire buffer.
>> >
>> > Consider the LaTeX math mode source, and the LaTeX environment name
>>
>> My question was, if you could point out problems with any of our bundled
>> backends (or, failing that, third-party ones). If they look okay, maybe
>
> Did some text get cut off here in your reply?
>
>> > source, and the LaTeX preamble source, etc. Trying to code the
>> > selection
>> > logic for all of these using the Company API looks very awkward to me,
>> > compared to the simplicity of setting a few buffer-local variables in
>> > Completion-UI.
>>
>> If performance is the problem, we could solve that by either:
>>
>> 1) Adding some pre-completion hook which would allow you to run some
>> code once, set a buffer-local variable, which all backend functions for
>> LaTeX can refer to later in `prefix' action.
>
> I don't think this can work, because the choice of source depends on the
> location of point and the text in the buffer, which changes from
> completion to completion.
>
> For the Predictive LaTeX support, I use a mish mash of piggy-backing on
> jit-lock where possible, some regexp match tests, and a limited form of
> incremental parsing of the LaTeX code when the rest aren't enough. (I
> used to exclusively use the latter, implemented using my auto-overlays
> package. Surprisingly this was more than efficient enough to cause no
> noticeable delay when typing and auto-completing, even on very old
> hardware. But writing and debugging the incremental parser definitions
> were a nightmare, largely because TeX doesn't have a well-defined
> grammar. So I replaced as much as possible of the parsing with simpler
> methods.)
>
> This kind of thing is simpler to do in proper languages (assuming you
> already have an incremental parser like CEDET, or a framework like
> nxml-mode).
>
>
>> 2) Create a special "merged" backend that would collect the results from
>> all LaTeX backend in a carefully defined fashion.
>
> This effectively means moving the selection logic out of Company and into
> Predictive. Which isn't necessarily a bad solution. It's how I used to do
> it in Predictive, until I generalised the selection logic and moved it
> into Completion-UI so other sources could make use of it.
>
> If we stick with the CAPF API, I suspect I'll end up moving my source
> selection logic back into Predictive, and making it Predictive-specific
> again.
>
>> 3) Indeed add some hook analogous to `auto-completion-source-functions'.
>> But yeah, CAPF already does that.
>>
>> > Sure, I could move that logic into Predictive itself, and have a single
>> > Predictive LaTeX backend. But that serves to demonstrate that the API
>> > isn't flexible enough to let me do what I want easily. Other markup
>> > languages and programming languages make similar demands on the API.
>>
>> We bundle several backends, some of them for programming modes, and so
>> far you haven't pointed out specific problems with any of them.
>
> Well, so far none of the existing Company backends are trying to do
> auto-completion from a dictionary of 100,000 words, in a language with no
> regular grammar, and where even tenths of a second lag are sufficient to
> make typing unusable.
>
> It would be an interesting experiment to stress-test the selection
> mechanism by reimplementing Predictive's LaTeX support in Company. But it
> would take some effort to code, and I don't have time (nor really the
> interest) to code this up, especially when the Emacs API for this is
> likely to look quite different.
>
>
>> >> Or you can define new backends that would do some common checks in
>> >> `prefix' (maybe calling an extracted function with common code) and
>> >> simply delegate all other actions to the respective base backend.
>> >> Implementing this is trivial.
>> >
>> > And once you've finished doing this, and factored out common selection
>> > mechanisms like regexps, faces and text properties into utilities
>>
>> Regexps we have already (company-grab-...),
>
> How fast is this if you have to go through, say, 100 moderately complex
> regexps checking for a match? (See the predictive-latex.el in the old
> Predictive tarball release for examples.)
>
>> instead of faces one should be using `syntax-ppss', font-lock isn't
>> always available, and text properties... hmm.
>
> Font-lock faces are *very* useful when you care about efficiency, because
> of all the effort that's gone into optimising jit-look. (Font-lock has
> very similar speed constraints to Predictive: it must not get in the way
> of typing.)
>
>> > functions...you'll have reimplemented something closer the
>> > Completion-UI
>> > API or c-a-p-f :)
>>
>> Only if we add a similar hook, see 3) above.
>>
>> >> I understand the principle, really. But the more one "cleanly
>> >> separates"
>> >> code, the harder it can be sometimes to read it, to get the full
>> >> picture.
>> >
>> > Indeed, which is why I listed grouping the completion and selection
>> > logic
>> > into one place as one of the things I liked about Company's API.
>> >
>> > Perhaps the cleanest and most flexible solution would be to have a list
>> > with entries of the form (TEST-FUNCTION . COMPLETION-FUNCTION),
>> > COMPLETION-FUNCTION is used if TEST-FUNCTION returns non-nil.
>>
>> I believe this suffers from the LaTeX problem you've described above. If
>> you have a dozen of completion functions for LaTeX, this scheme expects
>> you to have a dozen of corresponding test functions,
>
> Indeed.
>
>> and when one fails, the next will be called, and it won't be able to
>> use the results of the previous call (unless they set and use some
>> common buffer-local variable, which Company backends could also do; but
>> that's ugly).
>
> In my use-case, there's no useful information from previous tests anyway.
>
> But I don't think this is such an important use-case to consider for a
> generic API. Predictive has unusual requirements (always-on
> auto-completion as the main use-case imposes extreme efficiency demands).
> And (La)TeX and also Texinfo are odd cases, because their lack of a
> regular grammar makes background incremental parsing more difficult
> compared to programming languages.
>
> Probably the best solution is to move the fast-selection logic I
> implemented for use in LaTeX and Texinfo can back into Predictive.
>
>
>> > And then supply a bunch of standard utility functions for use as
>> > TEST-FUNCTION's, for testing regexp matches, faces, text-properties,
>> > etc.
>>
>> Like I described previously, such stand-alone tests probably won't be
>> very useful as values of this alist. One usually has to call several of
>> them to see if a completion-function is suitable.
>
> I meant to write "for use *in* test functions". Simple utility functions
> for testing regexps (which you already have), faces, etc. which can be
> combined to build a test function.
>
>
>> >> It makes certain amount of sense, although it looks like it could make
>> >> creating a "merged" completion function more difficult.
>> >
>> > I doubt it'll be insurmountable. Also, merged completion functions are
>> > a
>> > rather advanced feature that may not belong in core Emacs anyway
>> > (though
>> > it would be good if the API supported them without ugly hacks).
>>
>> It's a good feature enabling some kinds of backends that aren't usually
>> useful on their own (like Yasnippet). In my book, that's a good argument
>> to accept or reject an API.
>
> I'd be very happy to see a solution in Emacs if you can come up with one
> and convince people to merge it into core. As I said, adapting the
> Completion-UI code to a different completion source API shouldn't be
> difficult.
>
>> >> We'll see.
>> >
>> > Indeed, we'll see. Whilst I'd be happy to see the somewhat complicated
>> > and opaque c-a-p-f API replaced with something cleaner and simpler, I
>> > don't see us winning that argument.
>>
>> Personally, I'd probably be fine with c-a-p-f as long as it's powerful
>> enough. Moving to a less featureful API is likely out of the question,
>> but if it's demonstrated that c-a-p-f is fairly unsuitable for
>> implementing some features, I believe it would be a good reason to rule
>> it out.
>
> In Completion-UI, I implemented combined completion sources through a
> `completion-ui-combining-complete' wrapper function. (They're not
> functionally equivalent to your merged sources, but the requirements are
> somewhat similar.)
>
> Would using something like that to build merged CAPF functions be a
> solution for Company? I know this isn't particularly user-friendly if you
> want to allow users to easily define their own merged sources. I don't
> know if that's something people regularly do in Company.
>
>
> As I said, it's not me you need to persuade if you want to replace
> CAPF. I would be happy to see a nicer API than CAPF make its way into
> Emacs core. But I also understand why there will be resistance from Emacs
> devs to replacing an existing API unless there's a clear and pressing need.
>
>
>> >> But the general approach, while flexible on its surface, complicates
>> >> things if I intend to use any existing sources, written by third
>> >> parties. Because their authors are unlikely to have anticipated the
>> >> logic I'll add in my custom predicate function and to have written any
>> >> code in their packages I might use. Or, at least, that's considerably
>> >> less likely.
>> >
>> > I don't get your argument here. You have to wrap the third-party
>> > completion function in Company in order to code the appropriate backend
>> > selection logic. In Completion-UI, you put that code in an
>> > `auto-completion-source-function' instead, and probably don't need to
>>
>> Which code? There's likely to be none.
>>
>> Hence, more effort required on my part.
>>
>> Here I'm describing an organizational problem caused by an API. Not a
>> technical one.
>
> I think we probably agree. Not worth pursuing this particular discussion
> further really, since I'm not suggesting we use the Completion-UI API for
> source selection.
>
> There's one thing we should perhaps think a bit more about.
>
> Is it right to say that the majority of the Company backends are selected
> based on global properties of a buffer (e.g. major-modes)? This seems to
> be the case for the default `company-backends' list. In Completion-UI I
> was almost entirely focused on selecting backends based on local
> properties at different locations within a buffer (regexps, faces,
> syntax, etc.).
>
> How do you envisage supporting local source selection in the new (or
> enhanced CAPF) API? Would this kind of local completion source selection
> always be implemented within a single backend, like the Company CEDET
> backend? Or should there be a convenient way of supporting it in the
> generic source selection API?
>
> This kind of relates to your point about the ordering being important...
>
>
>> >> Like mentioned above, delegating the search for completions to an
>> >> existing backend is trivial. These are functions, and as such they are
>> >> stateless. Just call (other-backend 'candidates current-prefix).
>> >
>> > But this results is convoluted and somewhat confusing code. If the
>> > selection logic and completion function were separate, you wouldn't
>> > need
>> > to use such hacks.
>>
>> Looks straightforward to me. Function delegation is a rather simple
>> concept.
>>
>> We have examples of that in the Company frontends: take a look at
>> `company-pseudo-tooltip-unless-just-one-frontend' and
>> `company-preview-if-just-one-frontend'.
>>
>> `company-dabbrev-code' also delegates to `company-dabbrev', but it just
>> uses a public function from that backend's package, which is also a
>> valid approach.
>
> Following the code path is unnecessarily convoluted. I can't tell that
> company-dabbrev-code internally delegates to the company-dabbrev backend
> just from looking at `company-backends'. If it looked more like:
>
> ((use-dabbrev-code-p . dabbrev-completion)
>  (use-dabbrev-p . dabbrev-completion))
>
> it would be immediately obvious that these are two different ways of
> using the same source of completions.
>
>> > It understand why you're harping on about the limitations of the
>> > default hook entries
>>
>> I just can't see how you find them useful on their own. I'll shut up
>> now. :)
>
> Look at predictive-latex.el, which uses them on their own :)
>
> But I'll shut up too. It seems clear to me from our discussion that we
> should either use CAPF or something closer to Company for the completion
> source selection API.
>
> From a Completion-UI perspective, I don't much care what we
> use. Switching between completion sources was never the main focus of
> Completion-UI (the UI was the main focus). I should be able to adapt the
> code to whatever we settle on. Most of it's abstracted into accessor
> functions/macros internally, anyway.
>
>> > Indeed, though this is starting to look a lot like defining some
>> > standard
>> > hook functions for auto-completion-source-functions. As you make the
>> > company-backends API more flexible and convenient, you'll increasingly
>> > find yourself reimplementing equivalent functionality to that of
>> > c-a-p-f
>> > and the Completion-UI API. If were to start simplifying the
>> > Completion-UI
>> > API or c-a-p-f, it'll increasingly look more like the Company API.
>> > Maybe
>> > the sweet spot is in the middle?
>>
>> Maybe. I can certainly see myself adding a
>> auto-completion-source-functions analog in Company, as an advanced
>> feature.
>
> The question for now is, what should go in the generic Emacs API? Vanilla
> CAPF, perhaps with some additional standard PROPS properties? If the
> generic Emacs API can't replace company-backends in Company, then in my
> view the API is no good.
>
> You're the expert here :)
>
>
>> >> Actually, I was thinking about the former option. Let's define widgets
>> >> with an API in the usual sense, so it can be used by both
>> >> `completion-at-point' and external packages. We'll need this kind of
>> >> API
>> >> either way, in order to be able to write new widgets.
>> >
>> > I'm confused. `completion-at-point' will never use an API for
>> > *defining*
>> > new completion UI widgets (`completion-ui-register-interface'). It will
>> > need to be modified to *invoke* the new UI widgets.
>>
>> Here I'm thinking in terms of Company approach, I guess. The API for
>> defining a backend or frontend and the way it is used are the same: a
>> protocol consisting of messages and proper reactions or responses to
>> them.
>>
>> Thus, "widget has to respond to..." would be in the API for defining a
>> widget, and "widgets respond to..." would be in the usage API.
>> Essentially the same.
>
> Sounds pretty close to the Completion-UI widget API. A widget has to
> provide "function that responds to x", "function that responds to y".
>
>> If you like to add convenience macros, extra infrastructure, etc, they
>> may diverge, but we'll need an API a completion package can use easily
>> use anyway, not just a bunch of private functions that can change will
>> every release.
>
> Absolutely.
>
> The way I imagine it, completion packages like Company would rarely need
> to use the API for defining new widgets. (Unless they're defining custom
> widgets that are so package-specific they don't belong in core Emacs. But
> that seems like a rare case.)
>
> What they *will* all use is the API for *invoking* the completion
> interface, which we began to discuss below.
>
>> > Copying and adapting some of the code from `complete-in-buffer' into
>> > `completion-in-region' would suffice for that. Then
>> > `completion-in-region' would remain the generic Emacs API for
>> > displaying
>> > and selecting completion candidates (only now it would display them in
>> > a
>> > nicer interface).
>>
>> Maybe. At a first glance, `completion-in-region' will need access to
>> PROPS returned by completion functions, not just COLLECTIONS.
>
> Agreed. I figured we could add an additional PROPS optional argument, as
> this wouldn't break backwards compatibility...
>
>> >> Yes, but see above. Using 1. from Company would be the current next
>> >> step
>> >> toward integration, as I see it.
>> >
>> > Really? Why should we use the Company UI widget API and interfaces in
>> > Emacs, when the API is less flexible and less customizable than that
>> > the
>> > Completion-UI widgetAPI, and it implements fewer widgets?
>>
>> Sorry for being unclear. I meant the reverse: Company would use the new
>> widget API defined here, while still retaining its backends, at least
>> initially.
>
> Thanks, makes sense now.
>
>> Although while you don't own the popup widget, we do have one. Maybe
>> porting that code won't be too hard.
>
> Great! It would be a good test of the API too, to see how easy it is to
> port.
>
> Sounds like your equivalent of the "dynamic" interface is better, too
> (you avoid modifying the buffer text). I'd like to either port the
> Company version, or adapt the Completion-UI one to use the Company
> mechanism (whichever is simpler).
>
>
>> > Did you really mean that you wanted to throw Completion-UI in the
>> > garbage, rewrite Company yourself to be suitable as a generic Emacs
>> > completion API, rewrite/port the missing UIs and features, and persuade
>> > people to merge it into Emacs core? (If so, great! Let me know when
>> > you're done, and then I can stop maintaining Completion-UI and switch
>> > Predictive over to the new interface :-)
>>
>> I'll think about it. :) Not sure about the "persuade people to merge in
>> into Emacs core", though.
>
> Ah, but that's the hard part - the rest is easy ;-)
>
>> But you could help kick this process off my filing an issue describing
>> Company's backend API shortcomings. Is it just the lack of
>> `auto-completion-source-functions'? Non-prefix completion?
>
> Erm, I don't see any major shortcomings in the API. Probably that hasn't
> come across in the discussion :-)
>
> Minor ones that have come here were:
>
> - No support for non-prefix completion (but it looks easy enough to
>   extend the API).
>
> - Difficult (impossible?) to distinguish between "no completions
>   available" and "backend not applicable".
>
>
>> > Probably I misunderstood what you wrote.
>>
>> Yep. :)
>
> Shame! I was hoping you were volunteering to do all the work ;-)
>
>> > I only really have one strong requirement: if some form of
>> > auto-completion mode gets included in the Emacs completion UI, I will
>> > argue hard for it to be as powerful as Completion-UI's
>> > `auto-completion-mode'. (Take a look at `auto-completion-syntax-alist'
>> > and `auto-completion-override-syntax-alist', and at the way all the UI
>> > widgets are integrated.) Anything less powerful, and it will be useless
>> > to me for Predictive.
>>
>> I'll take a look, thanks.
>>
>> >> Customizing hooks is a tricky business. I believe the opaqueness of
>> >> c-a-p-f to the user is the main problem with the current API.
>> >
>> > But above you argued that backends and selection logic are supplied and
>> > configured by package authors, not by users!? In which case package
>> > authors can simply supply a c-a-p-f function, and users can add the
>> > functions they want to c-a-p-f. (I believe Stefan made a similar point
>> > earlier.)
>>
>> Customizing a hook is tricky for user. Try `M-x customize-variable RET
>> find-file-hook'. You'll only see the buffer-local value, and not the
>> global one. But `run-hooks' uses both if the local value includes `t',
>> which it usually does.
>
> I see the global value in Customize, even when I deliberately add a local
> hook function. Showing the global value in Customize seems the correct
> thing to do. Is that not what Emacs does? (I wouldn't know - I've never
> Customized a hook. I always use `add-hook' in my .emacs for that.)
>
>> Users can modify the hooks programmatically, of course, but that's a
>> step more difficult. And they'll also need to understand the values
>> already there, to be able to remove or rearrange them.
>>
>> We'll also should discourage lambdas there. At the moment, my
>> find-file-hook contains this beauty, courtesy of autorevert.el:
>>
>> #[nil "\302\301!\210\303\304 !8\211 \207"
>>    [buffer-file-name auto-revert-tail-pos make-local-variable 7
>> file-attributes]
>>    3]
>>
>> We should have less of that.
>
> :-)
>
>> > Don't get me wrong. I'm no fan of the complexity of c-a-p-f. But you
>> > seem to be arguing both ways.
>>
>> Of course I'm arguing toward a middle ground comfortable to me. So:
>> package authors deciding when their backend is suitable to use -- good;
>> users deciding the order of trying backends and their groupings -- also
>> good.
>
> OK, makes sense. If I understand correctly, CAPF is fine for letting
> users decide the order of trying backends. It's the groupings part that's
> less clear, right?
>
>
>> > Swings and roundabouts. I could always add a
>> > `completion-ui-major-mode-source' function that checks an alist, and
>> > then
>> > it would be:
>>
>> Yes, well, here you are discarding the "standard, tried-and-tested Emacs
>> mechanisms" of major mode hooks and buffer-local values. Which was
>> exactly my point.
>
> Now I'm confused. I thought that was *my* point ;-)
>
>
>> > - foo completion function
>> > - bar completion function
>> > - foo predicate function
>> > - bar predicate function
>> > - add two entries to the alist
>>
>> You forgot adding `completion-ui-major-mode-source' to the relevant list.
>>
>> > - code 1st bundled completion function and selection logic
>> > - code 2nd bundled completion function and selection logic (which
>> >    duplicates the logic in the 1st one, with different parameters)
>> >    ...
>> > - code 8th bundled completion function and selection logic (which
>> >    duplicates the logic in the last 7, with different parameters)
>>
>> Could that be a one-line function call, in each case with different
>> arguments? Aside from potential performance problems, it sounds rather
>> neat.
>
> Not really. The selection logic for each source involves a list of
> regexps, faces, etc. to test, and these are different for each
> source. Passing lists of hard-coded regexps and faces around would be
> ugly.
>
> I guess you could store the regexps etc. in configuration variables (one
> variable per source, or a single alist), and just pass an identifier as
> an argument.
>
>> > - add 8 entries to company-backends
>> > - arrange for package-specific sources to be added to company-backends
>> >    dynamically
>> >
>> > Contrast with Completion-UI's, which involves a *lot* less coding:
>> >
>> > - register 1st source (no need to code a new completion function, just
>> >    set one parameter when calling `completion-ui-register-source')
>>
>> You can likewise delegate to the same completion function in Company
>> backends. It's the same amount of code (1 line).
>
> But in Completion-UI, you don't even have to write the code - the macro
> does it for you :)
>
>> > It would be almost trivial to switch Completion-UI over to
>> > c-a-p-f. All I'd need to do to replicate the existing functionality
>> > is add a few additional properties to the PROPS return argument.
>>
>> Great. I'd like to see the patch. :)
>
> I'll start working on it as soon as we agree on the completion source
> API, and as soon as I have some spare time. (Which unfortunately isn't
> likely to be before the end of term here in Cambridge, as I have a
> lecture course to prepare from scratch. But the feature-freeze makes it
> less pressing for now.)
>
>
>> > Presumably I could always set a buffer-local `company-backends' from
>> > predictive-mode that only contains relevant backends if necessary.
>>
>> Yes, of course.
>>
>> > c-a-p-f is trivially customizable by users, as long as packages /
>> > Company
>> > supplies suitable c-a-p-f functions for the backends.
>>
>> When you say "trivially customizable", do you mean via the Customize
>> interface?
>
> I just meant that it's a straight list of functions, so it's easy for
> users to understand how to configure it.
>
> I take your point about configuring hooks via the customization interface
> being tricky. But at least configuring hooks is standard Emacs foo. I
> suspect most users who want to change the order of functions in CAPF will
> know how to set hooks.
>
> It would be nice to define a Customization interface that lists the
> available CAPF functions in a drop-down menu (whilst also allowing
> free-form entries, of course). If I find a way of porting the
> Completion-UI auto-updating customization interface magic to CAPF, this
> would be feasible...
>
>
>> > *Writing* c-a-p-f is non-trivial, but as you said earlier those
>> > are/should be supplied by package authors. Or have I misunderstood?
>>
>> I don't think I said that about c-a-p-f, but yes, they should be.
>
> Good, we're on the same page.
>
>> >>> 3. The most popular and useful "list-the-available-completions" UI is
>> >>>      popup.el. Is there any chance of getting copyright assignment
>> >>> for
>> >>>      this? Or will we be forced to code something equivalent from
>> >>> scratch?
>> >>
>> >> https://github.com/auto-complete/popup-el/issues/50
>> >
>> > Fingers crossed...!
>>
>> I wouldn't hold my breath: IME, getting a response from that crowd on
>> non-trivial issues is hard. Collecting copyright assignments could be
>> harder still.
>
> Shame. I guess we'll be recoding it then. At least it won't be from
> scratch, since you've already got something similar in Company.
>
>> P.S. My email client displays each new message from you as a separate
>> thread, possibly because your email address is timestamped, or maybe
>> because they don't contain a "References" header. Could you do something
>> about that?
>
> Strange. Mutt threads them fine for me. The ones I've checked all contain
> an "In-Reply-To" header. I haven't changed the mutt reply header
> settings, so presumably mutt doesn't add "References" headers by default?
>
> Perhaps your email client is ignoring the In-Reply-To header? (I somehow
> doubt the date stamps in the tagged addresses are the issue. Threads
> always contain messages from different senders, so it threading should
> work even if your mail client considers the changing dated addresses to
> be different senders.)
>
> All the best,
> Toby
> --
> Dr T. S. Cubitt
> Royal Society University Research Fellow
> and Fellow of Churchill College, Cambridge
> Centre for Quantum Information
> DAMTP, University of Cambridge
>
> email: address@hidden
> web:   www.dr-qubit.org
>
>