bug#3687: 23.1.50; inconsistency in multibyte eight-bit regexps [PATCH]

[Mattias Engdegård]

28 juni 2019 kl. 15.03 skrev Eli Zaretskii <eliz@gnu.org>:
> 
> However, I don't want to look at the patch before we discuss and agree
> on the principles.

A most sensible approach.

> 1. What do you mean by "raw bytes"?  Is #xab a raw byte or a Unicode
>    point U+00AB?  IOW, how do we distinguish, in a regexp, between a
>    raw byte and a character whose Unicode codepoint is that byte's
>    value?  And how does one go about concocting a regexp that matches
>    raw bytes in a unibyte or multibyte buffer or string?

Sorry, I should have been more clear. The terminology in the manual is a bit 
muddled; in this case I mean the characters (or whatever you prefer calling 
them) obtained with hex or octal escapes in the range 128-255, such as "\xff" 
or "\377", regardless of the string's type (unibyte or multibyte).

Unicode characters in the range 128-255 can be generated using the \u00HH or 
\U000000HH notations, or by just including them literally. They are distinct 
from raw bytes.

To match raw bytes, just write them. They are not special in regexp syntax and 
need no escaping.

> 2. What is meant by "ranges from ASCII to raw bytes"?  Which
>    characters are included in such ranges?

Ranges such as [A-\xb1] or [\000-\377], where the first endpoint is an ASCII 
character and the last endpoint is a raw byte as defined above. These should 
include all characters from the first endpoint up to and including ASCII 127, 
and all raw bytes from 128 to the last endpoint. This makes intuitive sense for 
unibyte strings where such an interval is contiguous in the underlying 
representation; extending them to multibyte is obvious.

In fact, the existing regexp engine already works this way; I didn't need to 
change that at all.

> 3. If ranges from non-ASCII characters to raw bytes make no sense,
>    how would one go about specifying a range that includes all the
>    characters and raw bytes supported by Emacs?

"[\x00-\U0010ffff\x80-\xff]"
"[^z-a]"
(rx anything)
etc.

>  . Raw bytes are internally treated as "characters" whose Unicode
>    codepoints are in the range [#x3fff00..#x3fffff].
>  . The internal representation of raw bytes in buffers and strings
>    uses 2-byte sequences that begin with #xc0 or #xc1.
>  . Emacs jumps through hoops to never expose the above internals to
>    th external world.  Thus, any encoding of a string with raw bytes
>    will convert them to their single-byte representation, where they
>    are indistinguishable from the characters which have the same
>    codepoints, and many operations other than encoding also
>    silently perform these conversions.

This is also my understanding. The patch does not expose the internal 
representation of raw bytes.