Re: Finding message by ID fails when message not visible

[Eric Abrahamsen]

Kai von Fintel <fintel@mit.edu> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Kai von Fintel <fintel@mit.edu> writes:
>>
>>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>>
>>>> Kai von Fintel <fintel@mit.edu> writes:
>>>>
>>>>> Kai von Fintel <fintel@mit.edu> writes:
>>>>>
>>>>>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>>>>>
>>>>>>> Kai von Fintel <fintel@mit.edu> writes:
>>>>>>>
>>>>>>>> I'm puzzled by the following behavior: if I search in an
>>>>>>>> nnimap group
>>>>>>>> for a message with a particular message-ID, the search fails
>>>>>>>> unless the
>>>>>>>> message is visible. If for example, only new messages are
>>>>>>>> shown, the
>>>>>>>> search for an old message with a particular ID fails but the
>>>>>>>> same
>>>>>>>> message is found when "/o" has made the old messages appear.
>>>>>>>>
>>>>>>>> This is relevant because I use links to gnus messages from
>>>>>>>> org-mode and
>>>>>>>> more often than not the link fails: I get taken to the nnimap
>>>>>>>> group that
>>>>>>>> contains the message, but I get the error "No such article
>>>>>>>> (may have
>>>>>>>> expired or been canceled)", even though the message is in fact
>>>>>>>> in the
>>>>>>>> group.
>>>>>>>>
>>>>>>>> I suppose org-gnus.el might have to be amended to work around
>>>>>>>> this, but
>>>>>>>> first I'd like to understand whether this is expected behavior
>>>>>>>> in gnus.
>>>>>>>>
>>>>>>>> Can anybody enlighten me or help me trouble-shoot this?
>>>>>>>
>>>>>>> No, it's not expected, and I haven't seen it before -- 
>>>>>>> sorry for the
>>>>>>> unhelpful reply! I just tried it (dovecot imap server), and it
>>>>>>> worked
>>>>>>> both with and without angle brackets on the ID (sometimes
>>>>>>> brackets can
>>>>>>> be an issue).
>>>>>>>
>>>>>>> To be sure everyone's on the same page, how exactly are you
>>>>>>> searching?
>>>>>>> If you're in the *Group* buffer and you hit "G G" on the group,
>>>>>>> and
>>>>>>> paste in the message ID, is the message found? If you're
>>>>>>> already in the
>>>>>>> *Summary* buffer, there isn't really any way to "search" as
>>>>>>> such,
>>>>>>> there's only "limiting" what messages are shown by some
>>>>>>> criteria -- in
>>>>>>> those cases it's true you won't be able to get to unseen
>>>>>>> messages.
>>>>>>>
>>>>>>> But that should never affect following links from org mode...
>>>>>>>
>>>>>>> Eric
>>>>>>
>>>>>> Thanks for the reply, Eric. More details then.
>>>>>>
>>>>>> 1. GG in the group buffer works (after a brief delay, displaying
>>>>>> "Opening server FastmailLocal" and then "Searching
>>>>>> nnimap+FastmailLocal:Archive...done").
>>>>>>
>>>>>> 2. M-^ ("Fetch article with id ...) in the summary buffer (a)
>>>>>> works when
>>>>>> the message is visible, (b) fails when the message isn't visible
>>>>>> (with
>>>>>> "No such article (may have expired or been canceled)").
>>>>>>
>>>>>> 3. When I follow the org link, say
>>>>>> "gnus:nnimap+FastmailLocal:Archive#56614DE9.5030300@upf.edu", I
>>>>>> do get
>>>>>> taken to the right nnimap group in gnus but the article is not
>>>>>> displayed
>>>>>> (again, with "No such article (may have expired or been
>>>>>> canceled)").
>>>>>>
>>>>>> One funny thing is that when the "search" fails (in both cases
>>>>>> 2b and
>>>>>> 3), the article summary actually is visible (see screenshot:
>>>>>> https://www.dropbox.com/s/b1b9mxi7bl0a51j/missing-article.jpg?dl=0)
>>>>>>
>>>>>> If 1 and 2 are as expected, then I guess the issue is with
>>>>>> org-gnus and
>>>>>> how it interacts with my IMAP set-up (offlineimap + dovecot +
>>>>>> nnimap)?
>>>>>>
>>>>>> -- Kai.
>>>>>
>>>>> Some more follow-up. I tried to understand the code in
>>>>> org-gnus.el and
>>>>> org-sum.el. It appears that the functions "org-gnus-open" and
>>>>> "org-gnus-follow-link" call the function
>>>>> "gnus-summary-goto-article"
>>>>> from gnus-sum.el. The latter is called with the optional
>>>>> arguments "nil"
>>>>> and "t". The last one is supposed to force that all articles are
>>>>> loaded,
>>>>> but that doesn't seem to happen.
>>>>>
>>>>> When I call (gnus-summary-goto-article "56614DE9.5030300@upf.edu"
>>>>> nil t)
>>>>> in the summary buffer of the relevant group, it finds the message
>>>>> when
>>>>> it is listed and doesn't find it if the buffer has only the first
>>>>> 200
>>>>> articles (loading all articles with "/o" then makes the function
>>>>> succeed).
>>>>>
>>>>> I added (gnus-large-newsgroup nil) to the group parameters, which
>>>>> then
>>>>> loads all articles when entering the group, and
>>>>> (gnus-summary-goto-article "56614DE9.5030300@upf.edu" nil t) then
>>>>> succeeds in the summary buffer.
>>>>>
>>>>> However, the gnus link still doesn't work from outside gnus. It's
>>>>> as if
>>>>> the FORCE optional argument that is given to
>>>>> "gnus-summary-goto-article"
>>>>> has no effect.
>>>>
>>>> I've got the same setup as you (Gnus and local dovecot), but
>>>> perhaps the
>>>> versions are different? Can you tell us your versions for Gnus and
>>>> Org?
>>>>
>>>> When I step through `gnus-summary-goto-article', searching for a
>>>> message
>>>> ID that is *not* currently display, the "force" argument never
>>>> even
>>>> comes into play. I end up in this branch:
>>>>
>>>> (if (and (stringp article)
>>>>           (string-match "@\\|%40" article))
>>>>      (gnus-summary-refer-article article)
>>>>
>>>> So then on to `gnus-summary-refer-article' (no force argument
>>>> used), and
>>>> in that function I end up in the "t" branch of the "cond" later
>>>> on, and
>>>> eventually into "(gnus-summary-insert-subject message-id)". Then
>>>> that
>>>> leads to "(gnus-read-header id)", and all is well.
>>>>
>>>> I know this is tedious, but would you mind stepping through with
>>>> edebug
>>>> and seeing where things fall apart? My off-the-cuff guess is that
>>>> you're
>>>> using an older version of Gnus where the nnimap backend isn't
>>>> quite all
>>>> put together...
>>>>
>>>> If you aren't comfortable with edebug let me know and I can
>>>> provide
>>>> exact instructions. It's a very useful thing to know!
>>>>
>>>> Eric
>>>
>>> #### System Info
>>> - OS: darwin
>>> - Emacs: 24.5.1
>>> - Gnus: v5.13
>>>
>>> Well, that was fun ... sort of. I found something I can tweak to
>>> make it
>>> work, but I'm not clear on why this issue arises.
>>>
>>> At the relevant point, the article id is
>>> "<56614DE9.5030300@upf.edu>"
>>> (the <> having been added by gnus-summary-refer-article) and the
>>> group
>>> is "nnimap+FastmailLocal:Archive".
>>>
>>> In `gnus-request-head (article group)', a funcall to
>>> `nnimap-request-head'
>>> is constructed. Because `gnus-override-method' at that point
>>> contains a long
>>> construct `(nnimap "FastmailLocal" ...)', the group argument of the
>>> call
>>> to nnimap-request-head is set to nil rather than to "Archive". I
>>> take
>>> it, the idea is that nnimap should be able to return the correct
>>> group
>>> on its own?
>>>
>>> The relevant code (in gnus-int.el) is
>>>
>>> (setq res (funcall head article
>>>                          (and (not gnus-override-method)
>>>                               (gnus-group-real-name group))
>>>        (nth 1 gnus-command-method)))
>>>
>>> Now, `nnimap-request-head (article nil "FastmailLocal")' returns
>>> "(nil .
>>> 277)". I believe the "nil" in that cons should be the group
>>> ("Archive").
>>> Maybe that's a problem in nnimap.el but looking at the code I don't
>>> see
>>> how it could ever return anything other than nil.
>>>
>>> From that point on, the group info seems lost and the message can't
>>> be
>>> targeted.
>>>
>>> If I revert the commit to gnus-int.el that introduced the override
>>> method thingy
>>> (http://git.gnus.org/cgit/gnus.git/commit/?id=8ba1cd0b96466a96265ec5336728519aa6030d83),
>>> to the following instead of what's above:
>>>
>>> (setq res (funcall head article (gnus-group-real-name group)
>>>                      (nth 1 gnus-command-method))))
>>>
>>> my messages get found because the actual group is passed on to
>>> nnimap-request-head.
>>>
>>> So, I have things working to a certain degree, but I'm utterly
>>> confused
>>> about why this is happening.
>>>
>>> Can you see from this what the problem is?
>>
>> My guess is it might just be time to update Gnus. Nnimap has
>> undergone a
>> lot of changes over the past couple of years, as has the rest of
>> Gnus,
>> and while the versioning system has always confused me, I'm pretty
>> sure
>> v5.13 is several years old. The code branches you seem to be
>> following
>> don't match up with what I'm seeing (in git master, aka Ma Gnus),
>> and
>> it's probably not worth trying to figure out exactly what's going
>> on. I
>> find Ma Gnus pretty stable -- would you be willing to run from git?
>>
>> Eric
>
> OK, I switched to Ma Gnus from git master (`gnu-version' now reports
> "Ma
> Gnus v0.14)").
>
> No luck, same issue. And reverting `gnus-request-head' in gnus-int.el
> to
> the version without gnus-override-method makes the links work. I will
> run through it all again with edebug (which unfortunately is
> super-slow
> here) over the holiday break and see whether I can determine more
> precisely where my installation fails. I'm very puzzled that your
> pretty
> much identical set-up works.

Well, let me know how that goes. I just stepped through a couple of
link-following processes, and can tell you that the call to
gnus-request-head/nnimap-request-head also returns a cons of (nil
. 12345) here. By that point we're already in the proper Summary buffer,
so I don't think it matters that the group name isn't in there.

I notice that gnus-summary-insert-subject actually gets called twice,
with the same "id" argument, and only inserts the correct article on the
second pass, but I haven't gone so far as to figure out why that
happens. In the let* construct at the top of the function, this branch:

(t
  (gnus-read-header id))

Returns a full header vector (with article number, and the various
Subject/From headers) for the article I'm looking for.

Anyway, there are a few more points of data to work with, hope you can
sort it out...

Eric