Re: Ispell and unibyte characters

[Agustin Martin]

On Fri, Apr 13, 2012 at 01:51:15PM -0400, Stefan Monnier wrote:
> > ("catala8"
> >      "[A-Za-z]" "[^A-Za-z]" "['\267-]" nil ("-B" "-d" "catalan") nil 
> > iso-8859-1)
> 
> > Unless emacs knows the encoding for \267 (middledot "·") it cannot decode it
> > properly. I prefer to not use UTF-8 here, because I want the entry to also 
> > be
> > useful for ispell (and also be XEmacs incompatible). The best approach here
> > seems to decode the otherchars regexp according to provided coding-system.
> 
> There's something I don't understand here:
> 
> If you want a middle dot, why don't you put a middle dot?
> I mean why write "['\267-]" rather than ['·-]?

The problem is that in a dictionary alist you can have dictionaries with
different unibyte encodings, if you happen to have two of that chars in
different encodings I'd expect problems.

I really should have gone in more detail about the system where I noticed
this, even if it is a bit Debian specific.

I noticed this problem in aspell catalan entry provided by Debian aspell-ca
package. In Debian for the different aspell {and ispell and hunspell}
dictionaries alists are created on dictionary installation and stored in a
file (for the curious /var/cache/dictionaries-common/emacsen-ispell-dicts.el). 
Some maintainers provide \xxx, some provide explicit chars in different
encodings, and all that info it put together in dict alist form in that file,
so it cannot be loaded with a given unique encoding but as 'raw-text, and
that implies loading as bytes rather than as chars.

> I think this is related to your saying "I prefer to not use UTF-8 here",
> but again I don't know what you mean by "use UTF-8", because using
> a middle dot character in the source file does not imply using UTF-8
> anywhere (the file can be saved in any encoding that includes the
> middle dot).
> 
> For me notations like \267 should be used exclusively to talk about
> *bytes*, not about *chars*.  So it might make sense to use those for
> things like matching particular bytes in [ia]spell's output, but it
> makes no sense to match chars in the buffer being spell-checked since
> the buffer does not contain bytes but chars.

That is why I want to decode those bytes into actual chars to be used in
spellchecking, and make sure that they are decoded from correct
coding-system. Otherwise if process coding-system is changed to UTF-8 and
that stays as bytes matching the wrong encoding things may not work well.

If there is a consensus that I should not go the decode- way for otherchars,
I will not commit that part. For Debian I can simply keep loading
emacsen-ispell-dicts.el as raw-text and do the decode- processing on its
contents, before they are passed to ispell.el through
`ispell-base-dicts-override-alist', so this last contains chars more that
bytes. I however think that is better to keep the decode- stuff for more
general use.

I will wait at least a couple of days before committing so is clear what to
do.

Thanks all for your comments,

-- 
Agustin