Re: On language-dependent defaults for character-folding

[Elias Mårtenson]

On 19 February 2016 at 18:09, Eli Zaretskii <address@hidden> wrote:

 

> The Unicode character decomposition was never meant to be used to provide a feature such as character
> folding in Emacs.

That's not true.  Canonical equivalence, which is encoded in canonical
decompositions, is a must for searching.  Otherwise, what looks the
same on display will not be found, and will look like a bug.  See the
example I gave with ñ and ñ (the latter one is 2 characters).

Of course you have to use the decomposition algorithms to ensure that the precomposed and decomposed variations of the same character compares equal.

This is, however, different from using the decomposition to to decompose a character and then using the base character as the thing to match against. The latter is what Emacs is doing today, as far as I understand.

 

2 and 3 are the same as we do already, AFAICT.  (Collation charts
describe ordering, which is irrelevant for searching; other than that,
you will see that Emacs already implements the data shown in
http://unicode.org/charts/collation/.)

The collation charts also describe equivalence. If you look at the latin collation chart for example (http://unicode.org/charts/collation/chart_Latin.html) you will see that the characters are grouped. These are the equivalences I'm referring to.

Now, I note that on these charts, U+0061 LATIN SMALL LETTER A and U+2C65 LATIN SMALL LETTER A WITH STROKE compares as different characters, and the latter does not have a decomposition. Should this also be addressed?

 

As for the locale-specific parts: using that will only DTRT if we
assume that the majority of searches are done in buffers holding text
in locale's language.  Is that a good assumption?

My opinion is that the default search behaviour should depend primarily on the locale of the entire Emacs session. I.e. the locale of the user starting the application. I'm not disagreeing that allowing a buffer-local locale override this behaviour is a good idea, but as a Swedish speaker I really see å, ä and a as completely separate things, even if the language of the buffer that I am editing happens to be English. The equivalence of these characters is the odd behaviour here, and the one that should be enabled explicitly.

Also, if I happen to be editing a Spanish document (I don't speak Spanish) I would find equivalence of ñ and n to be incredibly useful, even though Óscar would grind his teeth at it. :-)

We are talking
about a multilingual Emacs, in an age of global communications, where
you can have conversations with someone on the other side of the
world, or read text that combines several languages in the same
buffer.  Do we really want to go back to the l10n days, when there was
ever only one locale that was interesting -- the current one?  I
wonder.

Actually, I think so. This is because the search equivalence is inherently a local thing. The behaviour of search is more tried to a user's preference than the locale of the given buffer, in most cases.

At least that's my opinion. The bike shed can have many colours.

 

It is, Unicode provides it.  We just didn't import it yet.

It does? I was looking for such tables, but didn't find it. Do you have a link?

 

It's more complex than that, but patches are welcome, of course.

Having spent the better part of the day trying to solve a C++ design problem that I had originally hand-waved as being trivial, I know what you mean…

 

Note that the prerequisite for anything more complicated and elaborate
than what we have now is to re-implement character-folding on the C
level, inside search.c functions.  The current implementation is at
its limits already.  I tried to convince the interested people to do
this in C to be gin with, but couldn't, and the feature was important
enough to have even in its current implementation.

I'm not going to offer to do this until I'm sure that I can have the copyright assignment done. But I am interested in it.

Regards,

Elias