Unable to get all messages in an IMAP group

[Vincent Bernat]

Hey!

Since the latest IMAP changes, I am unable to enter some groups and on
some others, I don't get all the messages. For example, currently, I see
3 messages in my INBOX while there 10 when I use another IMAP client.

When I refresh the group, I get "nnimap read 0k from " with nothing else
after. What could be the problem?

[Steinar Bang]

Post by Vincent Bernat
Since the latest IMAP changes, I am unable to enter some groups and on
some others, I don't get all the messages. For example, currently, I see
3 messages in my INBOX while there 10 when I use another IMAP client.
When I refresh the group, I get "nnimap read 0k from " with nothing else
after. What could be the problem?

What server? Dovecot? Gmail? Other?
Do you use the agent on the nnimap groups where you have the issue?

[Vincent Bernat]

Post by Steinar Bang

Post by Vincent Bernat
Since the latest IMAP changes, I am unable to enter some groups and on
some others, I don't get all the messages. For example, currently, I see
3 messages in my INBOX while there 10 when I use another IMAP client.
When I refresh the group, I get "nnimap read 0k from " with nothing else
after. What could be the problem?

What server? Dovecot? Gmail? Other?
Do you use the agent on the nnimap groups where you have the issue?

It's Dovecot 2.2.13. I don't use the agent. I see that nnimap-debug
doesn't exist anymore. How to get some traces?

Also, the content of .newsrc.eld may be inexact. I am synchronizing it
across my hosts and I am using isync to replicate IMAP mailboxes (served
locally by Dovecot). So, each host has its own view of the IMAP
server. Usually, after a sync, M-g is enough to get Gnus and Dovecot
agree again.

[Vincent Bernat]

Post by Vincent Bernat

Post by Steinar Bang

Post by Vincent Bernat
Since the latest IMAP changes, I am unable to enter some groups and on
some others, I don't get all the messages. For example, currently, I see
3 messages in my INBOX while there 10 when I use another IMAP client.
When I refresh the group, I get "nnimap read 0k from " with nothing else
after. What could be the problem?

What server? Dovecot? Gmail? Other?
Do you use the agent on the nnimap groups where you have the issue?

It's Dovecot 2.2.13. I don't use the agent. I see that nnimap-debug
doesn't exist anymore. How to get some traces?
Also, the content of .newsrc.eld may be inexact. I am synchronizing it
across my hosts and I am using isync to replicate IMAP mailboxes (served
locally by Dovecot). So, each host has its own view of the IMAP
server. Usually, after a sync, M-g is enough to get Gnus and Dovecot
agree again.

I am using that in Gnus:

#v+
(setq gnus-select-method
;; Primary server: dovecot
`(nnimap ""
(nnimap-stream shell)
(nnimap-shell-program "/usr/lib/dovecot/imap -o mail_location=maildir:~/.mbsync/mails/luffy")
(nnir-search-engine imap)))
#v-

IMAP server is feeded through isync/mbsync. So, UUID or whatever IMAP
uses to identify messages may be off-sync. This never was a problem
before.

[Steinar Bang]

Post by Vincent Bernat
It's Dovecot 2.2.13. I don't use the agent.

Hm, ok... from your symptoms, I suspected the agent, but obviously that
wasn't the problem.

And dovecot is the best-behaved IMAP server nnimap is used
against.

Post by Vincent Bernat
I see that nnimap-debug doesn't exist anymore.

So it is.

Post by Vincent Bernat
How to get some traces?

I don't know, unfortunately.

Post by Vincent Bernat
Also, the content of .newsrc.eld may be inexact. I am synchronizing it
across my hosts and I am using isync to replicate IMAP mailboxes (served
locally by Dovecot). So, each host has its own view of the IMAP
server. Usually, after a sync, M-g is enough to get Gnus and Dovecot
agree again.

What happens if you move your .newsrc.eld aside and create a new
.newsrc.eld for the nnimap groups? Does it work initially and then get
corrupted? Does it work initially and continue to work?

[Tassilo Horn]

Post by Steinar Bang

Post by Vincent Bernat
I see that nnimap-debug doesn't exist anymore.

So it is.

Post by Vincent Bernat
How to get some traces?

I don't know, unfortunately.

,----[ C-h v nnimap-record-commands RET ]
| nnimap-record-commands is a variable defined in `nnimap.el'.
| Its value is nil
|
| This variable may be risky if used as a file-local variable.
|
| Documentation:
| If non-nil, log commands to the "*imap log*" buffer.
`----

Bye,
Tassilo

[Vincent Bernat]

Post by Steinar Bang

Post by Vincent Bernat
Also, the content of .newsrc.eld may be inexact. I am synchronizing it
across my hosts and I am using isync to replicate IMAP mailboxes (served
locally by Dovecot). So, each host has its own view of the IMAP
server. Usually, after a sync, M-g is enough to get Gnus and Dovecot
agree again.

What happens if you move your .newsrc.eld aside and create a new
.newsrc.eld for the nnimap groups? Does it work initially and then get
corrupted? Does it work initially and continue to work?

I'll try tonight. On my laptop, there is no such problem (same
newsrc.eld). It should be related to those out-of-sync UID.

[Steinar Bang]

Post by Vincent Bernat
I'll try tonight. On my laptop, there is no such problem (same
newsrc.eld). It should be related to those out-of-sync UID.

One other possibility:

Go back to the previous version of gnus in git, and if that works, use
git bisect to find the commit where the change happened.

(If going back to the previous version of gnus doesn't help, then this
approach won't work, obviously...)

[Vincent Bernat]

Post by Steinar Bang

Post by Vincent Bernat
I'll try tonight. On my laptop, there is no such problem (same
newsrc.eld). It should be related to those out-of-sync UID.

Go back to the previous version of gnus in git, and if that works, use
git bisect to find the commit where the change happened.
(If going back to the previous version of gnus doesn't help, then this
approach won't work, obviously...)

So, it's easy to fix by going back. I have bisected (and ensured to not
fix the situation by not saving .newsrc.eld) and found that the problem
is this commit:
https://github.com/emacsmirror/gnus/commit/447c56af7cb2afac14a0f3ed23f85d907c443a5d

Before this commit, if I enter the group, I don't see all the messages
but if I hit M-g on the group, then enter it, everything is here (which
is what I expect). With this commit, doing M-g still gives an incomplete
view of the group.

[Steinar Bang]

Post by Vincent Bernat
So, it's easy to fix by going back. I have bisected (and ensured to not
fix the situation by not saving .newsrc.eld) and found that the problem
https://github.com/emacsmirror/gnus/commit/447c56af7cb2afac14a0f3ed23f85d907c443a5d

Ah, the speedup-on-large-imap-move commit, I think...?
http://thread.gmane.org/gmane.emacs.gnus.general/85440

[Vincent Bernat]

Post by Steinar Bang

Post by Vincent Bernat
So, it's easy to fix by going back. I have bisected (and ensured to not
fix the situation by not saving .newsrc.eld) and found that the problem
https://github.com/emacsmirror/gnus/commit/447c56af7cb2afac14a0f3ed23f85d907c443a5d

Ah, the speedup-on-large-imap-move commit, I think...?
http://thread.gmane.org/gmane.emacs.gnus.general/85440

Yes.

Unfortunately, in my case, this is not fixed in today master. Before the
patch, M-g does:

#v+
21:46:17 [] 106 SELECT "INBOX"
21:46:17 [] 107 UID FETCH 1:* FLAGS
#v-

After the patch, it does:

#v+
21:44:23 [] 9626 EXAMINE "INBOX" (QRESYNC (1402209543 15465))
#v-

What easy way do I have to restore M-g behaviour to issue a full
refresh?

[Steinar Bang]

[snip!]

Post by Vincent Bernat
#v+
21:44:23 [] 9626 EXAMINE "INBOX" (QRESYNC (1402209543 15465))
#v-

Well, at least that explains why 'M-g' stopped working...

Post by Vincent Bernat
What easy way do I have to restore M-g behaviour to issue a full
refresh?

I don't know, unfortunately... hope that Lars get back online soon, with
an urge to fix Gnus issues...? (it sounded like he was going walkabout
in Australia the last time he posted)

[Vincent Bernat]

Post by Vincent Bernat

Post by Steinar Bang
Ah, the speedup-on-large-imap-move commit, I think...?
http://thread.gmane.org/gmane.emacs.gnus.general/85440

Yes.
Unfortunately, in my case, this is not fixed in today master. Before the
#v+
21:46:17 [] 106 SELECT "INBOX"
21:46:17 [] 107 UID FETCH 1:* FLAGS
#v-
#v+
21:44:23 [] 9626 EXAMINE "INBOX" (QRESYNC (1402209543 15465))
#v-
What easy way do I have to restore M-g behaviour to issue a full
refresh?

I am fixing that with this patch:

[Vincent Bernat]

Post by Vincent Bernat
Yes.
Unfortunately, in my case, this is not fixed in today master. Before the
#v+
21:46:17 [] 106 SELECT "INBOX"
21:46:17 [] 107 UID FETCH 1:* FLAGS
#v-
#v+
21:44:23 [] 9626 EXAMINE "INBOX" (QRESYNC (1402209543 15465))
#v-
What easy way do I have to restore M-g behaviour to issue a full
refresh?

This doesn't fix everything for me. I think there is another problem.

When I refresh my INBOX with M-g, then do g twice, I get this trace:

22:15:19 [] 4253 SELECT "INBOX"
22:15:19 [] 4254 UID FETCH 1:* FLAGS

Count is correct (all read).

22:15:28 [exoscale] 4260 EXAMINE "INBOX" (QRESYNC (1402209543 15465))

Count is correct (all read)

22:15:31 [exoscale] 4355 SELECT "INBOX"
22:15:31 [exoscale] 4356 UID FETCH 1:* FLAGS

Count is incorrect (many unread)

I suppose that M-g may not refresh the right server due to the empty
square brackets in contrast with what is happening with g. This doesn't
happen with my other INBOX (the primary one).

[Vincent Bernat]

Post by Vincent Bernat
This doesn't fix everything for me. I think there is another problem.
22:15:19 [] 4253 SELECT "INBOX"
22:15:19 [] 4254 UID FETCH 1:* FLAGS
Count is correct (all read).
22:15:28 [exoscale] 4260 EXAMINE "INBOX" (QRESYNC (1402209543 15465))
Count is correct (all read)
22:15:31 [exoscale] 4355 SELECT "INBOX"
22:15:31 [exoscale] 4356 UID FETCH 1:* FLAGS
Count is incorrect (many unread)
I suppose that M-g may not refresh the right server due to the empty
square brackets in contrast with what is happening with g. This doesn't
happen with my other INBOX (the primary one).

If I use this patch:

#v+
diff --git a/lisp/nnimap.el b/lisp/nnimap.el
index cdbc2a68709b..8f9afa634a87 100644
--- a/lisp/nnimap.el
+++ b/lisp/nnimap.el
@@ -822,6 +822,7 @@ textual parts.")

(deffoo nnimap-request-group-scan (group &optional server info)
(setq group (nnimap-decode-gnus-group group))
+ (nnimap-change-group group server)
(let (marks high low)
(with-current-buffer (nnimap-buffer)
(erase-buffer)
#v-

I get this trace:

22:33:33 [exoscale] 5936 SELECT "INBOX"
22:33:33 [exoscale] 5937 SELECT "INBOX"
22:33:33 [exoscale] 5938 UID FETCH 1:* FLAGS

So, it seems to use the right IMAP server but the count is the wrong one
(a lot of unread messages). But maybe the group is just borked, I could
use M-c on it. I leave it as is if it needs more debugging.

[Lars Ingebrigtsen]

Post by Vincent Bernat
+ (nnimap-change-group group server)

I've installed a version of this in git Gnus. It seems to be doing the
right thing for me now, at least...

[Vincent Bernat]

Post by Lars Ingebrigtsen

Post by Vincent Bernat
+ (nnimap-change-group group server)

I've installed a version of this in git Gnus. It seems to be doing the
right thing for me now, at least...

Thanks! It works fine for me now.

[Lars Ingebrigtsen]

From 025abeeebac0c196b8efa3aa538bcc69b7037374 Mon Sep 17 00:00:00 2001
Date: Wed, 4 Feb 2015 22:08:48 +0100
Subject: [PATCH] nnimap: fix group scan request
`gnus-group-get-new-news-this-group' was not refreshing the group with
the newly introduced `nnimap-request-group-scan' method due to two
naming problems.

Thanks; applied.