Re: Why multibyte for original-article buffer

[Stefan Monnier]

>> I've been running with the following patch with good results.
>> Also the patch makes sense to me: the original article is a sequence of
>> bytes (i.e. encoded chars), so a unibyte buffer makes a lot of sense
>> (it's more efficient and is likely to hide fewer bugs).

> So, it is better to make also the " *nntpd*" buffer, the process
> buffer for pop3, etc. be unibyte.  If such changes cause a problem,
> it will be due to a code that copies data from those buffers to
> a multibyte buffer and then decodes the data.  Actually, making
> the original article buffer be unibyte prevents me from reading
> a Japanese 8bit shift_jis message like this.

> 日本語

> (Try `g' in the summary buffer.)

Indeed, I see that.  The patch below will fix it.

>> Yet the code currently explicitly sets the buffer to multibyte mode.
>> Does anybody know why?

> IIRC, it was done about ten years ago.  I guess it was probably
> a workaround for a problem like the one mentioned above.

Yes, the Gnus code is full of such things.  I suggest it's time to take
a step back and think about how things *should* work.  Then change the
parts for which we know how they should work.  This will introduce bugs,
but then we'll know that the bugs are elsewhere and that fixing them
will get us closer to a good solution.


        Stefan


@@ -6564,7 +6564,13 @@
                 (with-current-buffer gnus-original-article-buffer
                   (and (equal (car gnus-original-article) group)
                        (eq (cdr gnus-original-article) article))))
-           (insert-buffer-substring gnus-original-article-buffer)
+            ;; `insert-buffer-substring' would incorrectly use the
+            ;; equivalent of string-make-multibyte which amounts to decoding
+            ;; with locale-coding-system, causing failure of
+            ;; subsequent decoding.
+            (insert (mm-string-to-multibyte
+                     (with-current-buffer gnus-original-article-buffer
+                       (buffer-substring (point-min) (point-max)))))
            'article)
           ;; Check the backlog.
           ((and gnus-keep-backlog