Re: On language-dependent defaults for character-folding

[Eli Zaretskii]

> Date: Fri, 19 Feb 2016 18:51:47 +0800
> From: Elias Mårtenson <address@hidden>
> Cc: Lars Ingebrigtsen <address@hidden>, emacs-devel <address@hidden>
> 
>  > The Unicode character decomposition was never meant to be used to provide 
> a feature such as
>  character
>  > folding in Emacs.
> 
>  That's not true. Canonical equivalence, which is encoded in canonical
>  decompositions, is a must for searching. Otherwise, what looks the
>  same on display will not be found, and will look like a bug. See the
>  example I gave with ñ and ñ (the latter one is 2 characters).
> 
> Of course you have to use the decomposition algorithms to ensure that the 
> precomposed and decomposed
> variations of the same character compares equal.

Then you agree that _some_ form of character-folding should be turned
on by default?

> This is, however, different from using the decomposition to to decompose a 
> character and then using the
> base character as the thing to match against. The latter is what Emacs is 
> doing today, as far as I understand.

Please describe in more detail why do you think what Emacs does today
is not what you think it should do.  It's possible we have a
miscommunication here.

For example, if the buffer includes ñ (2 characters), should "C-s n"
find the n in it?

>  2 and 3 are the same as we do already, AFAICT. (Collation charts
>  describe ordering, which is irrelevant for searching; other than that,
>  you will see that Emacs already implements the data shown in
>  http://unicode.org/charts/collation/.)
> 
> The collation charts also describe equivalence.

That equivalence is encoded in the decomposition data that is part of
UnicodeData.txt which Emacs uses for character-folding.

> If you look at the latin collation chart for example
> (http://unicode.org/charts/collation/chart_Latin.html) you will see that the 
> characters are grouped. These are
> the equivalences I'm referring to.

Yes.  And if you look at the entries of the equivalent characters in
UnicodeData.txt, you will see there they have decompositions, which is
what Emacs uses for searching when character-folding is in effect.

> Now, I note that on these charts, U+0061 LATIN SMALL LETTER A and U+2C65 
> LATIN SMALL LETTER A
> WITH STROKE compares as different characters, and the latter does not have a 
> decomposition. Should this
> also be addressed?

Maybe so, but given the controversy even about what we do now, which
is a subset, I'd doubt extending what we do now is a wise move.

>  As for the locale-specific parts: using that will only DTRT if we
>  assume that the majority of searches are done in buffers holding text
>  in locale's language. Is that a good assumption? 
> 
> My opinion is that the default search behaviour should depend primarily on 
> the locale of the entire Emacs
> session. I.e. the locale of the user starting the application. I'm not 
> disagreeing that allowing a buffer-local locale
> override this behaviour is a good idea, but as a Swedish speaker I really see 
> å, ä and a as completely
> separate things, even if the language of the buffer that I am editing happens 
> to be English. The equivalence of
> these characters is the odd behaviour here, and the one that should be 
> enabled explicitly.
> 
> Also, if I happen to be editing a Spanish document (I don't speak Spanish) I 
> would find equivalence of ñ and n
> to be incredibly useful, even though Óscar would grind his teeth at it. :-)

So you are in fact making two contradicting statements here.  Indeed,
the locale in which Emacs started says almost nothing about the
documents being edited, nor even about the user's preferences: it is
easy to imagine a user whose "native" locale is X starting Emacs in
another locale.

>  We are talking
>  about a multilingual Emacs, in an age of global communications, where
>  you can have conversations with someone on the other side of the
>  world, or read text that combines several languages in the same
>  buffer. Do we really want to go back to the l10n days, when there was
>  ever only one locale that was interesting -- the current one? I
>  wonder.
> 
> Actually, I think so. This is because the search equivalence is inherently a 
> local thing.

Being a multi-lingual environment, Emacs has no real notion of the
locale.

>  It is, Unicode provides it. We just didn't import it yet.
> 
> It does? I was looking for such tables, but didn't find it. Do you have a 
> link?

Look for DUCET and its tailoring data.  These should be a good
starting point:

  http://www.unicode.org/Public/UCA/latest/
  http://cldr.unicode.org/
  
>  Note that the prerequisite for anything more complicated and elaborate
>  than what we have now is to re-implement character-folding on the C
>  level, inside search.c functions. The current implementation is at
>  its limits already. I tried to convince the interested people to do
>  this in C to be gin with, but couldn't, and the feature was important
>  enough to have even in its current implementation.
> 
> I'm not going to offer to do this until I'm sure that I can have the 
> copyright assignment done. But I am
> interested in it.

Thanks.