bug#8055: 24.0.50; [nnmaildir/Gnus] please support in-filename Maildir flags

[Luca Capello]

Hi there!

NB, I (X-Debbugs-)Cc:ed all the people I know have dealt with this
    problem, sooner or later.  Given that I could not find any related
    bug in the GNU BTS, I thought you would have been interested, please
    forgive me if this is not the case.

To anyone who replies: please remove any cc: except the BTS and mine, if
not explicitly requested, TIA.

Disclaimer: I know this is quite an old version of emacs-snapshot, but
given that this bug is quite old, I wanted to report it with the oldest
version I had installed.  I will upgrade to the latest emacs-snasphot
package [1] in the next days.

[1] <http://emacs.naquadah.org>

I guess there is nothing new in the nnmaildir support in Gnus, at least
it seems that nothing has changed since Brian Nelson's rant on the
debian-user@ mailing list back in 2004-08-04 [2].  in Brian words:

  [nnamildir] uses its own, errr, "system".  In each Maildir directory,
  it creates a ".nnmaildir" directory, which in turn contains a "marks"
  directory, which in turn contains directories like "read", "reply",
  and "ticked", which in turn contain hard links to the original message
  files.

  For example, if a mail was marked as seen and replied, you would find
  a hard link in .nnmaildir/marks/read/1234 and a hard link in
  .nnmaildir/marks/reply/1234, both of which point to cur/1234.

[2] Message-ID: <87hdrivzuk.fsf@scabbers.bignachos.com>
    URL: 
<http://lists.debian.org/msgid-search/%3c87hdrivzuk.fsf%40scabbers.bignachos.com%3e>
    URL: 
<http://news.gmane.org/find-root.php?message_id=%3c87hdrivzuk.fsf%40scabbers.bignachos.com%3e>

Without even talking about file pollution, this is a no-op, especially
if like Brian (and myself, FWIW) you want to synchronize your Maildir
using OfflineIMAP [3].

[3] <http://nicolas33.github.com/offlineimap/>

Brian solved this *big* problem by writing his own script to synchronize
in-filename Maildir flags with the nnmaildir ones (his script is
included in the email above).  And people started using this script
directly from OfflineIMAP [4].

[4] <http://nakkaya.com/2010/04/10/using-offlineimap-with-gnus/>

Another solution is to install a local IMAP server [5][6], but I do not
want to install a *full* IMAP server when what I need is simply a
synchronized copy of my remote IMAP Maildir.  And the required space
counts (at least for me), on a clean Debian sid chroot:
=====
root@gismo:/# apt-get install dovecot-imapd
[...]
The following NEW packages will be installed:
  dovecot-common dovecot-imapd libgcrypt11 libgnutls26 libgpg-error0
  libgssapi-krb5-2 libk5crypto3 libkeyutils1 libkrb5-3 libkrb5support0
  libldap-2.4-2 libmysqlclient16 libpq5 libsasl2-2 libsasl2-modules
  libtasn1-3 mysql-common openssl ucf
0 upgraded, 19 newly installed, 0 to remove and 0 not upgraded.
Need to get 12.5 MB of archives.
After this operation, 27.7 MB of additional disk space will be used.
Do you want to continue [Y/n]? n
Abort.

root@gismo:/# apt-get install offlineimap
[...]
The following NEW packages will be installed:
  file libexpat1 libmagic1 mime-support offlineimap python
  python-minimal python-support python2.6 python2.6-minimal
0 upgraded, 10 newly installed, 0 to remove and 0 not upgraded.
Need to get 4763 kB of archives.
After this operation, 19.3 MB of additional disk space will be used.
Do you want to continue [Y/n]? n
Abort.
=====

[5] <http://www.emacswiki.org/emacs/JamesFerguson>
[6] 
<http://roland.entierement.nu/blog/2010/09/08/gnus-dovecot-offlineimap-search-a-howto.html>

Given that I am switching to a LUKS-encrypted SSD as my primary HD [7],
the number of files written by the current nnmaildir support in Gnus is
very much important.  Not saying anything about the impact on disk
space...  So I did some tests on my non-SSD HD [8] and the benchmark
method, AKA my .emacs:

--8<---------------cut here---------------start------------->8---
(setq debug-on-error t)
(setq gnus-select-method '(nnmaildir "local"
                                     (directory "~/Maildir/")
                                     (expire-age 'never)))

;;;; measure Gnus loading time
;;; <http://a-nickels-worth.blogspot.com/2007/11/effective-emacs.html>
(require 'cl)
(defun gismo-gnus ()
  (interactive)
  (let ((*gnus-load-start* (current-time)))
    (gnus)
    (message "My Gnus loaded in %ds"
             (destructuring-bind
              (hi lo ms)
              (current-time)
              (- (+ hi lo)
                 (+ (first *gnus-load-start*)
                    (second *gnus-load-start*)))))))
--8<---------------cut here---------------end--------------->8---

[7] actually, the HD was bought last September and because of this bug
    (as well as others) I have not completely switched yet :-(
[8] the default which came with my IBM/Lenovo ThinkPad X60
    <http://bugs.debian.org/406026>

After having imported the Maildir [9] in Gnus (keeping an unmodified
copy as ~/Maildir.BCK), I quit and compared the disk space and the
number of files in both Maildirs:

  $ du -s $MAILDIR/
  $ find $MAILDIR/ -type f | wc -l

[9] please note that this is *not* my whole Maildir (around 1400
    folders), but only a subset, still containing 747 folders, the most
    "heavy" ones...

I then re-iterated the test above at different times, always taking care
of having done other work in between, and the results are not at all
encouraging (Gnus vs. BCK):

1) the difference in disk space is constant, but significant,
   i.e. 1.093GB (6272120 - 5125812 = 1146308)

2) the difference in the number of files is constant as well, but now it
   becomes crazy, i.e. 3 times more (824905 - 274722 = 550183)

3) the time required to import the Maildir is way too much and not
   constant at all, given that the function above returned

     24113s   -38647s   14399s   -51172s   14145s   14224s

   I do not understand the reason for the negative values: I thought
   they were caused by the fact that I performed some tests over the
   night (and I completely forgot to also keep the timestamps), but
   given that `current-time' returns the seconds since the Unix epoch
   [10], crossing the midnight should not be a problem.

   Removing the two negative values above [11], the average is 16720s,
   more than 4.5 hours!  My "natural" observations agree with this
   value: I sometime started the test on purpose just before going to
   bed to be sure it will be finished the morning after [12].

4) the reload time, i.e. the time Gnus needed to start with an
   already-imported Maildir, is more constant, i.e. 1129s, around 19
   minutes (raw values are 1130s, 1229s, 1175s, 1089s and 1020s)

[10] <http://en.wikipedia.org/wiki/Unix_time>
[11] I am too lazy now to try to sort good values (and tired of this bug
     and all the tests as well)
[12] I usually sleep between 6 and 7 hours per night

Now that I know that it is for me impossible to use Gnus with Maildir, I
would like to help *in any way* to find a solution, which IMHO is quite
simple: supporting the in-filename Maildir flags.

I am not such a good programmer (I know a bit of Lisp, both the Common
and Emacs variants), however I have a strong motivation to not leave
Gnus.  I tried to live with Mutt for a while, but it was like learning
to walk again and there was nothing similar to the `*Group*' buffer, not
even talking about the default way Gnus shows emails, i.e. hiding the
already read (AKA "ancient") ones.

Thx, bye,
Gismo / Luca

If Emacs crashed, and you have the Emacs process in the gdb debugger,
please include the output from the following gdb commands:
    `bt full' and `xbacktrace'.
For information about debugging Emacs, please read the file
/usr/share/emacs/24.0.50/etc/DEBUG.


In GNU Emacs 24.0.50.1 (x86_64-pc-linux-gnu, GTK+ Version 2.20.1)
 of 2010-12-13 on elegiac, modified by Debian
 (emacs-snapshot package, version 1:20101212-2)
configured using `configure  '--build' 'x86_64-linux-gnu' '--host' 
'x86_64-linux-gnu' '--prefix=/usr' '--sharedstatedir=/var/lib' 
'--libexecdir=/usr/lib' '--localstatedir=/var' '--infodir=/usr/share/info' 
'--mandir=/usr/share/man' '--with-pop=yes' 
'--enable-locallisppath=/etc/emacs-snapshot:/etc/emacs:/usr/local/share/emacs/24.0.50/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/24.0.50/site-lisp:/usr/share/emacs/site-lisp'
 '--without-compress-info' '--with-x=yes' '--with-x-toolkit=gtk' 
'--with-imagemagick=yes' 'build_alias=x86_64-linux-gnu' 
'host_alias=x86_64-linux-gnu' 'CFLAGS=-DDEBIAN -DSITELOAD_PURESIZE_EXTRA=5000 
-g -O2' 'LDFLAGS=-g -Wl,--as-needed' 'CPPFLAGS=''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: en_US.UTF-8
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: en_US.UTF-8
  value of $LC_TIME: nil
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: nil
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

Major mode: Group

Minor modes in effect:
  gnus-undo-mode: t
  gpm-mouse-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
ESC x g i s m o - g n u TAB RET ESC x r e p o r t - 
e m a TAB RET

Recent messages:
Reading active file from local via nnmaildir...
Reading active file from archive via nnfolder...
Opening nnfolder server on archive...done
Reading active file from archive via nnfolder...done
747 new newsgroups have arrived
Checking new news...done
Setting up drafts group
Warning: Setting up drafts group
My Gnus loaded in 14224s
Auto-saving...

Load-path shadows:
/usr/share/emacs/24.0.50/site-lisp/auctex/tex-fold hides 
/usr/share/emacs/site-lisp/auctex/tex-fold
/usr/share/emacs/24.0.50/site-lisp/auctex/tex-jp hides 
/usr/share/emacs/site-lisp/auctex/tex-jp
/usr/share/emacs/24.0.50/site-lisp/auctex/tex-font hides 
/usr/share/emacs/site-lisp/auctex/tex-font
/usr/share/emacs/24.0.50/site-lisp/auctex/texmathp hides 
/usr/share/emacs/site-lisp/auctex/texmathp
/usr/share/emacs/24.0.50/site-lisp/auctex/toolbar-x hides 
/usr/share/emacs/site-lisp/auctex/toolbar-x
/usr/share/emacs/24.0.50/site-lisp/auctex/tex-buf hides 
/usr/share/emacs/site-lisp/auctex/tex-buf
/usr/share/emacs/24.0.50/site-lisp/auctex/tex-info hides 
/usr/share/emacs/site-lisp/auctex/tex-info
/usr/share/emacs/24.0.50/site-lisp/auctex/tex-mik hides 
/usr/share/emacs/site-lisp/auctex/tex-mik
/usr/share/emacs/24.0.50/site-lisp/auctex/context-nl hides 
/usr/share/emacs/site-lisp/auctex/context-nl
/usr/share/emacs/24.0.50/site-lisp/auctex/tex-fptex hides 
/usr/share/emacs/site-lisp/auctex/tex-fptex
/usr/share/emacs/24.0.50/site-lisp/auctex/multi-prompt hides 
/usr/share/emacs/site-lisp/auctex/multi-prompt
/usr/share/emacs/24.0.50/site-lisp/auctex/tex-style hides 
/usr/share/emacs/site-lisp/auctex/tex-style
/usr/share/emacs/24.0.50/site-lisp/auctex/font-latex hides 
/usr/share/emacs/site-lisp/auctex/font-latex
/usr/share/emacs/24.0.50/site-lisp/auctex/bib-cite hides 
/usr/share/emacs/site-lisp/auctex/bib-cite
/usr/share/emacs/24.0.50/site-lisp/auctex/tex hides 
/usr/share/emacs/site-lisp/auctex/tex
/usr/share/emacs/24.0.50/site-lisp/auctex/context hides 
/usr/share/emacs/site-lisp/auctex/context
/usr/share/emacs/24.0.50/site-lisp/auctex/tex-bar hides 
/usr/share/emacs/site-lisp/auctex/tex-bar
/usr/share/emacs/24.0.50/site-lisp/auctex/latex hides 
/usr/share/emacs/site-lisp/auctex/latex
/usr/share/emacs/24.0.50/site-lisp/auctex/context-en hides 
/usr/share/emacs/site-lisp/auctex/context-en
/usr/share/emacs-snapshot/site-lisp/psgml/psgml hides 
/usr/share/emacs/site-lisp/psgml/psgml
/usr/share/emacs-snapshot/site-lisp/psgml/psgml-other hides 
/usr/share/emacs/site-lisp/psgml/psgml-other
/usr/share/emacs-snapshot/site-lisp/psgml/psgml-init hides 
/usr/share/emacs/site-lisp/psgml/psgml-init
/usr/share/emacs-snapshot/site-lisp/psgml/psgml-sysdep hides 
/usr/share/emacs/site-lisp/psgml/psgml-sysdep
/usr/share/emacs-snapshot/site-lisp/psgml/psgml-html hides 
/usr/share/emacs/site-lisp/psgml/psgml-html
/usr/share/emacs/24.0.50/site-lisp/debian-startup hides 
/usr/share/emacs/site-lisp/debian-startup
/usr/local/share/emacs/site-lisp/fortune hides 
/usr/share/emacs/24.0.50/lisp/play/fortune
/usr/share/emacs-snapshot/site-lisp/emms/tq hides 
/usr/share/emacs/24.0.50/lisp/emacs-lisp/tq

Features:
(shadow sort mail-extr emacsbug nnfolder nndraft nnmh nnmaildir
gnus-agent gnus-srvr gnus-score score-mode nnvirtual gnus-msg gnus-art
mm-uu mml2015 epg-config mm-view smime password-cache dig mailcap nntp
proto-stream starttls tls gnus-cache nnir gnus-sum macroexp nnoo
gnus-group time-date gnus-undo nnmail mail-source format-spec gnus-start
gnus-spec gnus-int gnus-range message sendmail regexp-opt rfc822 mml
easymenu mml-sec mm-decode mm-bodies mm-encode mail-parse rfc2231
rfc2047 rfc2045 ietf-drums mailabbrev gmm-utils mailheader gnus-win gnus
gnus-ems nnheader gnus-util mail-utils mm-util mail-prsvr wid-edit
t-mouse cl planner-autoloads debian-el debian-el-loaddefs w3m-load
psgml-init muse-autoloads emms-auto emacs-goodies-el
emacs-goodies-custom emacs-goodies-loaddefs easy-mmode dpkg-dev-el
dpkg-dev-el-loaddefs bbdb-autoloads preview-latex tex-site auto-loads
tooltip ediff-hook vc-hooks lisp-float-type mwheel x-win x-dnd tool-bar
dnd fontset image fringe lisp-mode register page menu-bar rfn-eshadow
timer select scroll-bar mouse jit-lock font-lock syntax facemenu
font-core frame cham georgian utf-8-lang misc-lang vietnamese tibetan
thai tai-viet lao korean japanese hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese case-table epa-hook
jka-cmpr-hook help simple abbrev loaddefs button minibuffer faces
cus-face files text-properties overlay md5 base64 format env code-pages
mule custom widget hashtable-print-readable backquote
make-network-process dbusbind dynamic-setting font-render-setting
move-toolbar gtk x-toolkit x multi-tty emacs)

pgpZS693o_tvh.pgp
Description: PGP signature