RE: should locate-file-completion filter out "./" and "../"?

[Drew Adams]

    > 2. The duplicates *are* truly identical, from the perspective of
    > completion, which cares only about the key of alist TABLE
    > key-value pairs. AFAIK, both completing-read and read-file-name
    > completely ignore the values of such pairs.  They accept an alist,
    > but they use only the cars.

    I.e. even though they are not identical, the current code in
    Emacs doesn't
    pay attention to the difference in the value slot and thus mistakenly
    conflates such similar-but-not-identical entries.

Yes, on peut le dire comme ca. But my point in saying both that completions
are identical and they are not identical is that it depends on what is meant
by "completion". To critters like display-completion-list, completions are
strings, and if a user chooses a completion, then s?he gets a string.

It's possible to speak of the alist key-value pairs in the TABLE arg to
completing-read (and the symbols, if an obarray is used) as "completions",
but that is a different meaning. It is only in the sense of that different
meaning that the completions are not truly duplicates. And that is not the
meaning expressed in the doc etc. when "completions" are discussed.

    > 4. Other people who might like to take advantage of a)
    > duplicate keys with different values or b) the original candidate
    > order might not want to do everything that I do in my code.  They
    > might want to simply use `minibuffer-completion-help' to display
    > the candidates.  They will not be able to do that, because
    > `minibuffer-completion-help' eagerly and blindly
    > strips duplicates and sorts the candidates it displays.

    [ I'll ignore ordering here because I believe it's a separate issue ]

We're bound to have the same or a similar discussion about that issue. I
also need completion display to not mess with the order - or, rather, I need
to be able to control the order. I agree, however, that it's more or less a
separate issue. And again, I do my own display (I use
`display-completion-list', however), and I sort when I need to.

    So you're saying that minibuffer-completion-help is incorrect in
    that it should not remove such non-identical duplicates.

Yes, I said that before. I added that it doesn't bother me personally,
because I don't use `minibuffer-completion-help'.

    I think this makes sense, although only if the value slot is actually
    used somewhere. I.e. only if it gets changed such that the value slot
    gets propagated to display-completion-list.

If that happens, fine. But even if that never happens, let's leave open the
possibility. If `minibuffer-completion-help' sorts and strips, then code
(like the recent `locate-file-completion' will eventually come to depend on
that behavior. Better to cut the cord now, and say that callers of
`completing-read' should do their own sorting and duplicate removal. But see
below - the real issue is being able to control sorting and duplicates,
however that control is provided.

    > 5. *Unlike* `minibuffer-completion-help', the function that calls
    > completing-read or read-file-name *knows* whether it wants duplicates
    > displayed in *Completions* or not.

    Depends which kind of duplicate (identical or not), right?  Can you give
    an example?

When my search function calls `completing-read', it knows that it needs the
completion TABLE alist to be in the order of the regexp matches in the
buffer, and it knows that it doesn't want to remove any TABLE entries that
might have the same key. Therefore, it passes an alist TABLE arg to
`completing-read' that contains all hits, in buffer order (actually, see
below on the order).

On the other hand, a function of mine that calls `completing-read' for file
names knows that it wants to present the file-name candidates in alphabetic
order in *Completions*, and without any duplicates (which there would be
only if the candidate files names were relative and were taken from more
than one directory).

Actually, I lied a bit, by omission. Because I want to let the caller
determine sorting, and a caller has no way to sort the candidates ahead of
time when an obarray is used for `completing-read', my completion code
always sorts, using a variable sort function, `icicle-sort-function'. To
inhibit sorting, the caller binds this to nil.

FYI - I let users define their own sort function (`icicle-sort-function' is
customizable), and I let them define an alternative sort function, for use
dynamically via `M-,' (in the minibuffer). The default value of the
alternative sort function sorts candidates in two alphabetic groups: first,
those candidates that have been used previously, then, those that have never
been used. So, by default, users get alphabetic sort, and, if they use
`M-,', they get historical inputs first and unused candidates second (each
group sorted alphabetically).

The point here is that the function `minibuffer-completion-help',
responsible for posting the completion candidates, should not have any say
in whether candidates are sorted or duplicates are removed. One way or
another, that should be up to the function calling `completing-read',
because only that function knows the semantics involved for the user.

    > That function should have the responsibility of removing
    > duplicates, if it deems that appropriate.  Likewise sort.
    > It should have the possibility of not removing duplicates
    > or not sorting, and have that reflected in the
    > *Completions* list. This seems logical to me -
    > `minibuffer-completion-help' should not be deciding what the
    > caller needs.

    minibuffer-completion-help doesn't work in a vaccum: it uses
    data provided by the caller.  So we can easily make it possible
    for the caller to tell minibuffer-completion-help whether to
    remove duplicates or not, and whether to sort or not.

Precisely. `minibuffer-completion-help' shouldn't have any say in the
matter, whether or not it is the executioner. The way to do that, as I see
it, is for the caller to send the list of possible candidates to
`completing-read', and for `completing-read' to display them with no
duplicate removal.

As I said, sorting needs to be dealt with differently from simply giving
`completing-read' the candidates in the right order, because of the
inability (or undesirability) of presorting an obarray (it could be
converted to a sorted list, but the sorted case is the usual one, and we
don't want the performance hit. My solution of binding the variable that
holds the sort function is, I think, a good approach here. The same approach
could be used for duplicate removal, though I haven't done that.

    > 6. Yes, in Emacs it might seem silly now for
    > `locate-file-completion' to remove duplicates, because
    > they are always removed anyway by `minibuffer-completion-help'.
    > I'm suggesting to remove that recent change to
    > `minibuffer-completion-help', and let the caller send just
    > what it needs to completing-read. Having the caller strip and
    > sort is cleaner, in the sense that it is lazier, and it leaves
    > open the possibility that Emacs too, or other users, might
    > someday make some use of the values in key-value pairs.

    Removing duplicates and sorting is the right thing to do in 99% of the
    cases,

Yes.

    so it makes a lot of sense to do it once and for all, rather than
    dump that responsibility onto every single caller.

(I think you mean "in one place", rather than "once and for all", right?)

No. It is naturally the responsibility of the caller. We can make duplicate
removal and sorting the default case, but it is the caller that must decide
whether to override the default.

If you like, duplicate removal could be treated the way I treat sorting:
Have a transform function that always massages the candidates to be
displayed. By default, this function would remove duplicates. The function
would be the value of a variable (a la `icicle-sort-function'). If bound to
nil, then no transfomation would be made: displayed candidates = input
candidate set (modulo subsequent sorting by the sort function).

It would also be possible to combine sorting and duplicate removal in a
single function, but I think it might be simpler to separate them (two
variables).

    If you want a way to turn off that sorting and duplicate removal,
    that's fine, but doing it by default is TRT.

You're right.

Another concern I have, before we go off and define the best way to do that
for Emacs, is that I want my code to work with older Emacs versions. That's
not your concern, I know, but it is a concern of mine. Obviously, if the new
mechanism could be easily reconciled with how things worked before, that
would make life easier for me. The use of function variables that I
described would allow that. Anyway, this is a relatively minor point, as I'm
sure I'll need to treat the new way differently, to some extent.

    > 7. Regardless of whether you are convinced that, even for
    > Emacs itself, the proper place for duplicate removal is at the point
    > of call, this makes a difference to me, because, unlike
    > `minibuffer-completion-help', I don't assume that *all* functions
    > calling completing-read want to strip dups and
    > sort candidates. In my code, it is the function that calls
    > completing-read that strips and sorts, if appropriate.

    (with-provocative-mode
     So you suggest to change the way Emacs works rather than
     change the way your code works?)

Well sure ;-). It was the recent change to `minibuffer-completion-help' that
broke my code.

But the reason I suggested what I did is that I think that change is the
wrong approach. Right now, `minibuffer-completion-help' steps in and sorts
and strips whether you like it or not. The responsibility rests naturally
with the caller, because only the caller knows the semantics of the
interaction and the candidate completions. And I do agree with you that it
is enough if the caller can inhibit the default behavior of sorting and
stripping.

With the proviso that it would be better to be able to specify a different
sort, rather than just inhibit sorting. Two mise-en-forme variables would be
good: `completion-transform-function' and `completion-sort-function'. The
former would do <whatever> to the set of candidates to be displayed - the
result it returns would be the list of candidates sent to the sort function.
The sort function would sort the list of candidates it receives and pass the
result to the display routine. Again, the two functions could be combined
into a single filter function.

In sum, let the caller grab the list of completion candidates before it is
displayed, do something to it (filter etc.), and return the actual list of
candidates to display. Let the caller bind transform or sort variable to nil
to inhibit the default behavior, or bind them to other functions to replace
that behavior.

Binding function variables seems to me like the right way to do these
things. I didn't use a variable for the duplicate-stripping because I didn't
need to, but now that I think of it, that would be a clean approach, and I
might change my code to do that.

BTW, I also give users some additional control over the list of completion
candidates, letting them 1) add extra candidates and 2) apply global
filtering operations.

1. The extra (pseudo)candidates are completions they always want included.
These are appended to the list of completions normally available (which are
based on the user's input). Programmers can bind this, for example to make
sure certain buffers are always choosable candidates, even when they don't
match the user's input.

2. The global filtering operations are applied to the universal set of
candidates (according to the type of input), before the user's input is
matched against it. This lets programmers customize how an existing function
operates. The function blindly passes its own predicate to
`completing-read', but the programmer's predicate (if any) is applied first.
There is not only a predicate variable, but also two regexp variables, for
additional filtering (keep and flush).

What all of this amounts to is giving more control over the list of
completion candidates to programmers.

    > I only really hope to persuade you about #7 (as well as to
    > please continue to not strip and sort in all-completions and
    > display-completion list).

    There has never been any intention to strip duplicates and/or sort in
    all-completions or in display-completion-list, AFAICT.

I know; I'm just paranoid ;-).

    > I probably can't convince you that Emacs or other users might
    > want to do something like I do, but perhaps I can convince you
    > that something like stripping and sorting should be done on an
    > on-demand basis, remaining agnostic on the question as long as
    > possible, and letting the code at point of call decide on the
    > behavior.

    If the strip&sort is done by the caller (as you suggest) before
    passing the resulting list to minibuffer-completion-help, then
    there's no way for someone to use his own version of
    minibuffer-completion-help which doesn't sort and doesn't strip.

    I.e. doing it in minibuffer-completion-help (controlled by an
    option that can be set by the caller) is exactly what you ask for:
    remaining agnostic on the question as long as possible.

If it is the caller of `completing-read' that has the (only) control, then
I'll be a happy camper. Where things get done is irrelevant to me, as long
as I can control it.

Thx - Drew