bug#17130: 24.4.50; Deficient Unicode case folding

[Nathan Trapuzzano]

Eli Zaretskii <eliz@gnu.org> writes:

>> > So you would need to search all characters to find those which have σ
>> > in the CANONICALIZE slot -- not very efficient, to say the least.
>> 
>> Doesn't this already happen?
>
> No, not when that slot is used for case-insensitive search.  You just
> use it to get the canonical equivalent, i.e. use the one-way mapping
> that it provides.

I still don't get it.  What I say below may explain why.

>> If not, then what is the CANONICALIZE slot doing that couldn't be
>> done with the regular upcase/downcase slots by themselves?
>
> If that slot is "trivial", i.e. contains the lower-case variant of the
> character, then indeed this slot doesn't add information, I think,
> only utility.  But it doesn't have to contain the lower-case variant.

I know.  But if Emacs doesn't do Unicode folding, what is there other
than lower/upper variants?

>> > IOW, what you suggest will provide a one-way mapping, whereas we need
>> > a two-way mapping.
>> 
>> Not sure I follow.  Seems to me the CANONICALIZE slot is sufficient, at
>> least in principle.
>
> It is sufficient for mapping a character to its canonical equivalent,
> but not finding the non-canonical variants of a canonical character.
> IOW, it is not well suited to finding ς given just σ.

Finding the non-canonical variants is not something that happens (at
least in principle) during case-insensitive matching.  You convert both
the matching string and the string being matched into their canonical
equivalents and see if they match.  You never UNfold.  Case folding is
by definition a one-way operation.

>> That said, making M-l downcase terminal Σ to ς would be a nice feature
>> that could be enabled, e.g., by enabling a minor mode or by modifying
>> some *-functions variable of functions that get called before the normal
>> behavior of M-l is applied, etc.  But it shouldn't have anything to do
>> with Unicode-compliant case-insensitive searching.
>
> For searching, you only need the CANONICALIZE slot.  But what about
> replacing the search string while keeping the letter case in the
> replacement?  For that, CANONICALIZE alone is not enough, you need the
> reverse mapping.

There is no reverse mapping when it comes to folding.  There can't be,
since multiple characters can fold into the same character.

I don't fully understand what "case-replace" does (e.g. case being a
property of characters and not strings, what does it mean to "preserve
case" when replacing a string of length x with a string of length y
where x != y), but I don't think Unicode folding would complicate it.
There are three cases in Unicode: lower, upper, and title.  Upper and
title already overlap for the vast majority of codepoints, so there you
already have problems with a case-preserving replace.  That said "fold"
is not a case in Unicode; it's a one-way mapping of non-overlapping sets
of characters to a canonical equivalent, so it makes no sense to talk
about preserving case with respect to case folding.

Notandum: I was wrong about Unicode saying nothing about character
ordering for non-combining characters.  The "special casing" document
(ftp://ftp.unicode.org/Public/UCD/latest/ucd/SpecialCasing.txt) contains
context- and language- dependent case rules for certain characters,
including final sigma.  Notably, the document says that Σ in terminal
position should (or "may"--I'm not really sure about how to interpret
the document) downcase to ς.  That said, the document has _nothing_ to
do with case _folding_, which is always context- and language-
independent.

Rightly interpreted, therefore, case _conversion_ (such as in
case-preserving replace) and case-insensitive _searching_ (i.e. case
folding), according to Unicode, are orthogonal.  We don't have to
address both at the same time.

>> Given the point about ß, you're probably right.  Unless we can make
>> entries in the CANONICALIZE slot be strings rather than code points.
>
> This is Lisp; a vector slot can contain any Lisp object.  But using
> CANONICALIZE for what you want would be wrong, I think, because it
> will screw up case-insensitive search, which expects to find there a
> single character.

Right, that's what I meant.  Putting strings there would break
something.