bug#4593: 23.1; RMAIL bug regarding Message not valid RFC2822 message

[Lane A. Hemaspaandra]

I've since the upgrade to emacs23 been having a problem that when I read 
a mailfile sometimes it says Message is not a valid RFC2822 message
and I end up in a deeply corrupted state:  one can move up and down
in the header buffer, but the message window doesn't change messages,
and if one says the file one gets a somewhat mbox-like file that 
is not a valid mbox file that has the content of just 3 or 4 of 
the, say, hundred emails that had been originally in the mailfile.

I finally found a set of files on which I can precisely reproduce the
problem.  So if I do this:

(each "read" means i ran rmail-input on the named file)
read ~/junk
q
read ~/=shortCutToOldEmail/WESTFIELD
(at this point it says: Replacing BABYL format with mbox format...done)
q
read ~/WESTFIELD
q
read ~/junk
q
read ~/=shortCutToOldEmail/WESTFIELD

then bang, it says:
    Message is not a valid RFC2822 message
and shows as the first line of the screen the cryptic text beween the ===
lines

===
_SIZE_1000_LESS 0, BODY_SIZE_2000_LESS 0, BODY_SIZE_5000_LESS 0, 
BODY_SIZE_7000_LESS 0, BODY_SIZE_800_899 0, TO_NO_NAME 0, __CP_URI_IN_BODY 0, 
__FRAUD_419_ANTIABUSE 0, __FRAUD_419_CONTACT_NAME 0, __HAS_MSGID 0, 
__MIME_TEXT_ONLY 0, __SANE_MSGID 0, __STOCK_PHRASE_24 0, __TO_MALFORMED_2 0'
X-RMAIL-ATTRIBUTES: --------

This is a confirmation that your Westfield order was successfully received and
[and so on for the rest of the message]
===

and when i do this with set-variable debug-on-error set to true, that
gives this at this point:

===
Debugger entered--Lisp error: (error "Message is not a valid RFC2822 message")
  signal(error ("Message is not a valid RFC2822 message"))
  error("Message is not a valid RFC2822 message")
  rmail-error-bad-format()
  rmail-get-header-1("X-RMAIL-ATTRIBUTES")
  apply(rmail-get-header-1 "X-RMAIL-ATTRIBUTES")
  rmail-apply-in-message(1 rmail-get-header-1 "X-RMAIL-ATTRIBUTES")
  rmail-get-header("X-RMAIL-ATTRIBUTES" 1)
  rmail-message-attr-p(1 "......U")
  rmail-first-unseen-message()
  byte-code("^Hq\210    \204\f^@\304\305 !\210\n\203^S^@\306 \210\307 
\210^K\203^^^@\310\311!\210\304\207" [mail-buf msg-shown rmail-display-summary 
run-mail-hook rmail-show-message rmail-first-unseen-message rmail-summary 
rmail-construct-io-menu run-hooks rmail-mode-hook] 2)
  rmail("~/=shortCutToOldEmail/WESTFIELD")
  rmail-input("~/=shortCutToOldEmail/WESTFIELD")
  call-interactively(rmail-input t nil)
  execute-extended-command(nil)
  call-interactively(execute-extended-command nil nil)
===

Note by the way that it does this NOT the first time in that session 
that it rmail-input'ed the file ~/=shortCutToOldEmail/WESTFIELD
but rather on the second, and at a time when  it had another 
buffer WESTFIELD<2> of almost the same name.  (variations on 
my sequence that course this seem not to cause the bug.  i have 
no idea why; i'm sorry.)

and if one then types "h" for header mode, it says:
Message 1 is not a valid RFC2822 message

and if one saves the file one ends up with a file that is much smaller
than the (Babyl) what one started with---wildly shorter.  and that new
file itself is sort-of-mbox-format but not into a valid mbox format.
It started at size 978312 and it ends up as size 41363.  (note to
myself: i have saved this file under the name
=shortCutToOldEmail/junksample.)

I have saved away the files:
 ~/junk
 ~/=shortCutToOldEmail/WESTFIELD
 ~/WESTFIELD
 ~/=shortCutToOldEmail/junksample
so that I have copies of all the files involved, in case that would 
be helpful. 

As a final comment, let me mention a few things.  When the above happens, 
 ~/=shortCutToOldEmail/WESTFIELD
starts in Babyl format.  Also, I did the above with NO .emacs file.  
When I have in place my .emacs file, which does set some variables
such as 
 (setq rmail-display-summary t)  ; always display summary
I get, as above, an error the second time I read 
 ~/=shortCutToOldEmail/WESTFIELD
but when I then hit "h" for header mode, I get a different 
behavior than above, namely, it says in the message line
on the bottom of the screen:
 Error in post-command-hook: (error Message is not a valid RFC2822 message)
and what it displays on the screen is a message but starting with 
the following lines (except without the leading ">" marks), and yes,
the first and third are identical:

===
>X-RMAIL-ATTRIBUTES: --------
>From Notify@westfieldcomics.com  Tue Jul 14 20:15:00 2009
>X-RMAIL-ATTRIBUTES: --------
>X-Coding-System: undecided-unix
>Return-Path: <Notify@westfieldcomics.com>
>X-Spam-Checker-Version: SpamAssassin 3.2.5-cs.rochester.edu_001 (2008-06-10) 
>        on slate.cs.rochester.edu
>X-Spam-Level: 
>X-Spam-Status: No, score=-13.3 required=5.0 tests=AWL,BAYES_60,
>        USER_IN_WHITELIST autolearn=no version=3.2.5-cs.rochester.edu_001
>X-Spam-Pyzor: 
>X-Spam-Report:                                                         
>        *  -15 USER_IN_WHITELIST From: address is in the user's white-list
>        *  1.0 BAYES_60 BODY: Bayesian spam probability is 60 to 80%
>        *      [score: 0.6241]
>        *  0.7 AWL AWL: From: address is in the auto white-list
===

I've been using RMAIL for many, many years, but I don't understand 
well how emacs/rmail actually work.  But I'll be very happy to do my
best to provide any additional info or run any tests that might help
diagnose this.  (And my apologies if it is some problem due to me.  
But the mailfile it is choking on is a typical babyl file.  And I 
have this problem repeatedly, with other files too, but not too
often.)  

Thank you very, very much.

Sincerely,
Lane


In GNU Emacs 23.1.1 (i386-redhat-linux-gnu, GTK+ Version 2.16.6)
 of 2009-09-22 on x86-6.fedora.phx.redhat.com
configured using `configure  '--build=i386-redhat-linux-gnu' 
'--host=i386-redhat-linux-gnu' '--target=i586-redhat-linux-gnu' 
'--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' 
'--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' 
'--includedir=/usr/include' '--libdir=/usr/lib' '--libexecdir=/usr/libexec' 
'--localstatedir=/var' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' 
'--infodir=/usr/share/info' '--with-dbus' '--with-gif' '--with-jpeg' 
'--with-png' '--with-rsvg' '--with-tiff' '--with-xft' '--with-xpm' 
'--with-x-toolkit=gtk' 'build_alias=i386-redhat-linux-gnu' 
'host_alias=i386-redhat-linux-gnu' 'target_alias=i586-redhat-linux-gnu' 
'CFLAGS=-DMAIL_USE_LOCKF -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 
-fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i586 
-mtune=generic -fasynchronous-unwind-tables''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: C
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: nil
  locale-coding-system: utf-8-unix
  default-enable-multibyte-characters: t

Major mode: RMAIL

Minor modes in effect:
  shell-dirtrack-mode: t
  tooltip-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  global-auto-composition-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
ESC [ > 0 ; 2 4 2 ; 0 c ESC x r m a i l - i n SPC RET 
j u n k RET y q ESC x r m a i l - i n SPC RET = s h 
TAB W E S T F TAB RET q ESC x r m a i l - i n SPC RET 
W E S T F SPC DEL TAB RET q ESC x r m a i l - i n SPC 
RET j u n k RET q ESC x r m a i l - i n SPC RET = s 
h TAB W E S T F SPC DEL TAB RET ESC x r e p o r t - 
e m a c s - b u g RET

Recent messages:
(No changes need to be saved)
Wrote /tmp/rmail4600dlq
Writing messages to /tmp/rmail4600FDD...done
Replacing BABYL format with mbox format...
Marking buffer unmodified to avoid rewriting Babyl file as mbox file
Counting messages...done
Replacing BABYL format with mbox format...done
(No changes need to be saved)
Counting messages...done
(No changes need to be saved) [2 times]
rmail-error-bad-format: Message is not a valid RFC2822 message