Re: Single quotes in Info

[Eli Zaretskii]

> Date: Tue, 27 Jan 2015 23:15:22 -0200
> From: Artur Malabarba <address@hidden>
> Cc: emacs-devel <address@hidden>
> 
> Eli, if I may ask, did you get a chance to see the code? (it's quite short)
> The last couple emails give me the impression we're not quite on the same 
> page.

I did just now, and I don't think I was on a different page.

> The purpose of this is to allow the user to search for complex characters 
> (such as curly quotes or any of these ＂“””„⹂〞‟‟❞❝❠“„〝〟🙷🙶🙸) by typing a simple 
> character available on simple keyboards (such as the plain double quote "). 

But that's exactly where it falls short of supporting a more general
feature, which allows to find text that is "equivalent" to the one you
search for.  The limitation to "simple characters available on simple
keyboards" might seem a no-brainer for predominantly ASCII text, but
it _is_ a serious limitation for any non-ASCII script, certainly for
complex scripts, which Emacs supports for years.

> Each simple character, needs an entry on the `isearch-groups-alist' variable. 
> The max number of entries we'll ever need on this alist (in the very worst 
> possible scenario) is the number of simple characters in a simple keyboard 
> (which is way less than 5000 last I checked).

You seem to forget that modern keyboards and input methods support
much more than what meets the eye on the keyboard.  Even Latin locales
provide non-ASCII characters such as á and å.  It is also not uncommon
to copy/paste a search string from some text, in which case the search
string could include the "complex" characters, but you'd still want to
find their "simple" equivalents; your code, which transforms only the
search string, cannot support this use case.  Moreover, CJK locales
use input methods that can produce thousands of characters, and for
people in those cultures such input is "simple" because they can use
nothing simpler.

Using a database that maps ASCII characters to regexps doesn't scale
for supporting these use cases.  It doesn't even scale to the
above-mentioned Latin characters, because á has a sequence of 2
characters "a ́" as its canonical decomposition, so when I type á, I
expect to find both á and "a ́", and vice versa.  More complex scripts
have several forms of the same letter, such as the "final" form used
in Arabic and Hebrew for the last letter in a word -- typing one of
these forms should find any other form.  Etc. etc. -- there's a huge
complexity behind all this, and we need to support it if we want to be
respected as a text editor.

The way to support this is similar to how we support case-insensitive
search: we "fold" each character, both in the search string and in the
text being searched, using case tables, and then compare the "folded"
characters.  Similarly, to support equivalence, we need to produce a
canonical/equivalent decomposition from each character on both sides
of the comparison, and then compare the results.

As I said before, we already have all the necessary data in the
'decomposition' property of each character, we just need to use it in
a way that is similar to case tables, just slightly more complex
(because we are no longer talking single characters).

> > > Does it relate a simple character to all its complex
> > > equivalents? Or does it relate each complex character to a simple 
> > > alternative?
> > The latter.  Read paragraph 1.1 of UAX #15 for the starting point, and
> > also section 3.7 of the Unicode Standard.
> If it's the latter, then it's the wrong way for us to do an automated 
> approach. What we need is to know the whole set of Unicode characters which 
> is equivalent to a given ASCII character. Of course we can build this table 
> from the Unicode Standard (that's exactly what the `isearch-groups-alist' 
> variable is meant to do), I'm just saying an automated approach probably 
> isn't viable here.

I don't see why it won't be viable, or maybe I don't understand what
you mean by "automated" here.  I certainly don't think we should limit
ourselves to "simple characters", not for something as general-purpose
as text search.  This might be okay for Info only, but not if we want
it in isearch.el.

My idea is to use the 'decomposition' property to decompose each
character in the search string and in the text being searched, when
they need to be compared.  Exactly like we do with case-folding.