Re: Broken charset=utf-16be articles with Gnus and Emacs 21.3

[Reiner Steib]

On Mon, Mar 31 2003, Jesper Harder wrote:

> FWIW, Oort Gnus and current CVS Emacs works as expected.  It used to
> exhibit the same bug you're describing, 

Yes, I remember seeing this on the Ding-List in some of Kai's
articles.  But Kai and me didn't recall whether the problem was in
Emacs CVS-HEAD or in Oort Gnus (and whether it was fixed or not).

> but I was fixed earlier this year -- probably by this change:
>
> 2003-01-03  Dave Love  <fx@gnu.org>
>
>       * international/mule-cmds.el (sort-coding-systems):
>       Adjust priority of utf-16 and x-ctext.

Yes, that's it, thanks for pointing this out.  After applying the
following patch to lisp/international/mule-cmds.el from Emacs 21.3.1
and evaluation `sort-coding-systems', I get utf-8 as expected.

--8<---------------cut here---------------start------------->8---
--- mule-cmds.el        26 Dec 2002 17:27:20 -0000      1.216
+++ mule-cmds.el        3 Jan 2003 20:16:11 -0000       1.217
@@ -425,9 +425,18 @@
                    (let ((base (coding-system-base x)))
                      (+ (if (eq base most-preferred) 64 0)
                         (let ((mime (coding-system-get base 'mime-charset)))
+                          ;; Prefer coding systems corresponding to a
+                          ;; MIME charset.
                           (if mime
-                              (if (string-match "^x-" (symbol-name mime))
-                                  16 32)
+                              ;; Lower utf-16 priority so that we
+                              ;; normally prefer utf-8 to it, and put
+                              ;; x-ctext below that.
+                              (cond ((or (eq base 'mule-utf-16-le)
+                                         (eq base 'mule-utf-16-be))
+                                     16)
+                                    ((string-match "^x-" (symbol-name mime))
+                                     8)
+                                    (t 32))
                             0))
                         (if (memq base lang-preferred) 8 0)
                         (if (string-match "-with-esc$" (symbol-name base))
--8<---------------cut here---------------end--------------->8---

> Didn't it get applied to Emacs 21.3?

No.  If the EMACS_21_1_RC branch is still maintained, it should
probably be applied there as well.

On Mon, Mar 31 2003, Kenichi Handa wrote:

> Oops, I've just found that Emacs' coding systems utf-16-le and
> utf-16-be produce BOM (Byte Order Mark) which is a bug according to
> their specifications.  I've just installed a fix.

Does it make sense to apply it to EMACS_21_1_RC too?

>> Expected behavior:
>> - The article should be encoded with
>>   "Content-Type: text/plain; charset=utf-8".
> I don't know why GNUS prefers utf-16-X to utf-8.  At least,
> sort-coding-systems prefers utf-8.

Apparently not in Emacs 21.3, unless I misunderstood the
abovementioned patch to `sort-coding-systems'.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo--- PGP key available via WWW   http://rsteib.home.pages.de/