Re: Character folding in the pretest

[Eli Zaretskii]

> Date: Sat, 6 Feb 2016 10:37:06 +0100
> From: Per Starbäck <address@hidden>
> Cc: "Dirk-Jan C. Binnema" <address@hidden>,
>       Drew Adams <address@hidden>, emacs-devel <address@hidden>
> 
> From the opposers it has been argued as if this is something
> mandated by Unicode, so we can do nothing about it but to follow.

No one said anything like that.  The references to the Unicode
Standard and its various data and TRs are to make the point that the
feature as implemented is based on sound principles and not on some
arbitrary criteria.

No one said the feature is "mandated" in any way, shape or form.
Whether the features should be turned on by default is a matter only
we the Emacs community will decide.

> It doesn't matter if the result is seen as buggy or dumb by
> users. "This feature is simply folding as specified by the Unicode
> standard".

The Unicode Standard specifies _how_ to fold during search.  It also
includes recommendations _when_ to fold.  It doesn't mandate anything,
and even if it did, we don't need to heed to that.  Your arguments in
this part are a red herring.

> That is not so. Of course the Unicode Consortium is well aware of the
> issues that I, Oscar and others are pointing out, and that I'm sure
> Artur is well aware of.

We are all aware of that, please give us credit that we know something
about the issues involved.  It is you who seems to misunderstands
important aspects of this, see below.

> Eli Zaretskii:
> > Perhaps you aren't familiar with Unicode equivalence, in which case I
> > suggest these sources:
> >
> >   http://unicode.org/reports/tr10/#Searching
> >   http://www.unicode.org/notes/tn5/
> >   http://www.unicode.org/reports/tr30/tr30-4.html
> 
> But of course these take up issues like we have mentioned here. The
> first one mentions the aa/å equivalence in Danish for example. And to
> quote the last one:
> 
> #  In the general case, different search term foldings are applied for
> #  different languages. For example, accent distinctions are ignorable
> #  for some languages, but not for others. In English the accent in
> #  words like naïve is optional, while to a Swedish user 'o' and 'ö'
> #  are distinct letters.

It seems that you have read only the parts that confirm your views in
your eyes, and skipped or dismissed the rest.  And now you are
spreading your misunderstanding among others.

The facts are different.  Unicode indeed recognizes that different
languages change the rules to some degree.  However, it defines
several distinct degrees of conformance, and what we have now is the
lowest possible level of conformance, the one that is not tailored to
any particular language.  See Section 3.8 of TR#10, referenced above,
and Table 13 there.  What we in fact implemented is the default
collation weights, which are independent of language tailoring.

This is similar to the data we use for case-folding: it doesn't
include any language-specific tailoring, and so in some cases, like
Turkish dotless i issue, produces results that are incorrect in the
context of some specific languages.  Still we use it, and it generally
works very well.

In the long run, we should add language-specific tailoring to this and
other similar features.  Currently, we lack the infrastructure for
doing that in a useful way, so this further development must wait.
But it doesn't mean the feature isn't useful as it is now, and several
participants in this thread explicitly said they like what the feature
gives them.  Which doesn't surprise me, because it matches the advice
in the Unicode Standard, so I know we are on the right path.

> That is by the way the last draft of a withdrawn tecnical report.

(So why are you quoting from it and claim that it supports your POV?
If it's indeed a useless, withdrawn draft, then it has no relevance at
all, right?  Please decide whether you want to treat that report
seriously or not, and please be consistent with your decision.  Trying
to have the cake and also eat it doesn't add credibility to your
opinions.)

>   Draft UTR #30: Unicode Character Foldings has been withdrawn. It was
>   never formally approved; the last public version was a draft
>   UTR,which can be found at
>   http://www.unicode.org/reports/tr30/tr30-4.html.

Actually, that draft was mentioned because it includes interesting and
important stuff not mentioned in one place in any other publication I
know of.  I referred to it under an assumption that the reader will be
keenly interested in learning as much relevant background information
about the subject as possible, even if the report itself never made it
to the official status.

> We have to break out of the circles this is going in.

There are no circles.  We wanted to collect feedback, and we are
collecting it.  The pretest is going on for merely one week, and the
feedback we have already is useful, and it keeps coming in.  Stopping
that and making the decision now makes no sense to me.  The release is
still quite far away, and we have nothing to lose by hearing from more
people.  Assuming we want to make an informed decision, there's no
rush.

> Please John, put your foot down and don't let this continue ad
> infinitum.

No one intends to continue "ad infinitum".  That's another red
herring.  We should continue collecting feedback for a couple more of
pretest releases, that's all.  Then we can make the decision based on
that feedback.  I counted 10 people (excluding myself and Artur) who
expressed their clear opinions in this thread; that is way too few for
an intelligent decision, IMO.

> The options we have are instead:
> 
> (1) Let the default be as searching has worked before. Nothing gets
> worse for anyone.
> 
> We'll the start of a new exciting feature available, that will be just
> right for many users, and that will be tried by a lot others as well,
> giving feedback for the continued development that Artur has written
> that he already is planning.
> 
> (2) Make the fundamental feature searching work fundamentally
> different out of the box in a way that for many users will be seen as
> neat, and for many users will be seen as "buggy, dumb or completely
> oblivious to" the user's culture.

With all due respect, I don't think this is an objective description
of the alternatives.