Re: regex and case-fold-search problem

[Stefan Monnier]

> While working on emacs-unicode, I noticed a very difficult
> problem which also exists in the current emacs.
> 
> (let ((case-fold-search nil))
>   (string-match "[Þ-ß]" "Þ")) => 0
> (let ((case-fold-search nil))
>   (string-match "[Þß]" "Þ")) => 0
> 
> (let ((case-fold-search t))
>   (string-match "[Þ-ß]" "Þ")) => nil !!!
> (let ((case-fold-search t))
>   (string-match "[Þß]" "Þ")) => 0
> 
> When you see the output of M-x list-charset-chars RET
> latin-iso8859-1 RET,  you'll soon find what's going on.
> 
> The relevan character codes are as follows:
>       Þ (#x8DE)
>       ß (#x8DF)
>       (downcase ?Þ) == ?þ (#x8FE)
>       (downcase ?ß) == ?ß (#x8DF)
> 
> This problem is not specific to non-ASCII chars, it's just
> rarer to face such a sitution in ASCII chars.
> 
> (let ((case-fold-search nil))
>   (string-match "[A-_]" "A")) => 0
> (let ((case-fold-search t))
>   (string-match "[A-_]" "A")) => nil
> (let ((case-fold-search t))
>   (string-match "[A_]" "A")) => 0
> 
> In my opinion, specifying ranges by chars are nonsense
> because there should be no semantics in the order of
> characters codes.

Indeed.  POSIX basically says the behavior is unclear (it's locale-dependent).

But I think that if it works with (case-fold-search nil) it should
also work with (case-fold-search t).  The current behavior is really
counter-intuitive.

> But, anyway, we have to decide what to do.
> 
> (1) Regard the above case as a bug, and fix it completely.
>     As we don't support a range striding over different
>     charsets by the current Emacs, I think the fix is
>     difficult but not that much.  But, in emacs-unicode, we
>     can't have such a restriction, and thus the fix is very
>     difficult.

For ASCII it's pretty easy to fix.  But for other charsets, it's
indeed more tricky.  Maybe we can simply use the smallest contiguous
range of chars that includes all the chars we should match,
so the behavior is indeed "implementation-defined" (in the sense
that it's not necessarily obvious to the user what happens) but
it's at least less confusing (in the sense that (case-fold-search t)
matches at least as much as (case-fold-search nil)).

> (2) Regard the above case as an (unpleasant) feature, and
>     document it.

I think we should document the fact that char-ranges shouldn't
be relied upon too much, especially outside of ASCII.  That's
true no matter how we deal with the problem.

> (3) Signal an error for such a regex (and of course document it).

That might be an option as well.


        Stefan