Re: Emacs trunk and Gnus master are fully sync'd now

[Giorgos Keramidas]

On Tue, 09 Jul 2013 21:33:24 +0200, David Engster <address@hidden> wrote:
> Giorgos Keramidas writes:
>> So the unexist ranges *are* there.  But as you said the information
>> seems to be lost somewhere between the time `.newsrc.eld' loads and the
>> time Gnus queries the imap server for group state.
>
> Yes, and I found the problem. I could not reproduce this initially
> because it only exists in Emacs trunk, not in Gnus git. After I switched
> to Emacs trunk, I saw the same thing.
>
> It's actually really simple: The problem is `gnus-clean-old-newsrc',
> where 'unexist' marks will be removed when you upgrade from an older
> Gnus version, probably because the format has changed. However, in Emacs
> trunk, `gnus-newsrc-file-version' is "Gnus v5.13", which is always
> considered *smaller* than "Ma Gnus v0.03", so it will always remove the
> marks:
>
>     (when (or force
>             (< (gnus-continuum-version gnus-newsrc-file-version)
>                (gnus-continuum-version "Ma Gnus v0.03")))
>       ;; Remove old `exist' marks from old nnimap groups.
>
> So this has to be changed or maybe completely removed. I'm not sure; I
> guess it *could* happen that someone running Ma Gnus v0.02 upgrades to
> Gnus from Emacs 24.4.

Great!  Now we know at least why this is triggered.  Looking at the
diffs between emacs-24 and trunk of Emacs I can see that this function
indeed only exists in the trunk:

% 0709 21:49 address@hidden:~/git/emacs/lisp/gnus$ git diff emacs-24..trunk -- 
gnus-start.el
% [...]
% @@ -2291,7 +2301,27 @@ If FORCE is non-nil, the .newsrc file is read."
%           (gnus-message 5 "Reading %s...done" newsrc-file)))
%
%        ;; Convert old to new.
% -      (gnus-convert-old-newsrc))))
% +      (gnus-convert-old-newsrc)
% +      (gnus-clean-old-newsrc))))
% +
% +(defun gnus-clean-old-newsrc (&optional force)
% +  (when gnus-newsrc-file-version
% +    ;; Remove totally bogus `unexists' entries.  The name is
% +    ;; `unexist'.
% +    (dolist (info (cdr gnus-newsrc-alist))
% +      (let ((exist (assoc 'unexists (gnus-info-marks info))))
% +       (when exist
% +         (gnus-info-set-marks
% +          info (delete exist (gnus-info-marks info))))))
% +    (when (or force
% +             (< (gnus-continuum-version gnus-newsrc-file-version)
% +                (gnus-continuum-version "Ma Gnus v0.03")))
% +      ;; Remove old `exist' marks from old nnimap groups.
% +      (dolist (info (cdr gnus-newsrc-alist))
% +       (let ((exist (assoc 'unexist (gnus-info-marks info))))
% +         (when exist
% +           (gnus-info-set-marks
% +            info (delete exist (gnus-info-marks info)))))))))
%
%  (defun gnus-convert-old-newsrc ()
%    "Convert old newsrc formats into the current format, if needed."

If I'm reading this function right, then _every_ time someone runs Gnus
from the trunk of Emacs they would have to do an 'initial sync' of all
their groups, because "Gnus v5.13" will never be >= "Ma Gnus.*".

Doing an initial sync for large sets of IMAP groups means that the users
will have to download several MBytes of data _every time_ they start
Gnus.  We should probably fix this some way toa void so large startup
downloads and, as a result, slow startup times.

The time different during the startup of Gnus and the amount of data
downloaded is too large to ignore it.  Running the following snippet in
emacs-24 and trunk I got the following results a few minutes ago:

(let ((info (format "%s"
                    (list
                     (format-time-string "%H:%M:%S Loading Gnus" (current-time))
                     (gnus)
                     (format-time-string "%H:%M:%S Finished loading Gnus" 
(current-time))))))
  (with-current-buffer "*scratch*"
    (insert info)))

Output:

| Branch          | Start Time | End Time | Duration | Downloaded |
|-----------------+------------+----------+----------+------------|
| Emacs-24 branch |   22:03:24 | 22:03:45 | 21 sec   | 148k       |
| trunk branch    |   22:04:43 | 22:06:11 | 88 sec   | 18350k     |

A 4x delay in startup time and more than 10x difference in downloaded
data is really _huge_.