Recent nnir update broke search on office365

[Dave Goldberg]

All searches come up empty since this commit (per git-bisect)

046825b09ff6e23ec7dfacd451d10985615b9793 is the first bad commit
commit 046825b09ff6e23ec7dfacd451d10985615b9793
Author: Eric Abrahamsen <***@ericabrahamsen.net>
Date: Tue Jan 27 16:35:25 2015 +1100

* nnir.el (nnir-run-imap): Enable non-ASCII IMAP searches.

:040000 040000 5131b4d4212b134d58fdaec4481a41239a69926c
e22212e758aa6a15e13c23ec8fb00f65bcf3a741 M lisp

I should have thought to check my other server (cyrus) - I can do so when I get back to work next week. I'm happy to help troubleshoot but need guidance on what to look for.

Thanks,

[Eric Abrahamsen]

Post by Dave Goldberg
All searches come up empty since this commit (per git-bisect)
046825b09ff6e23ec7dfacd451d10985615b9793 is the first bad commit
commit 046825b09ff6e23ec7dfacd451d10985615b9793
Date: Tue Jan 27 16:35:25 2015 +1100
* nnir.el (nnir-run-imap): Enable non-ASCII IMAP searches.
:040000 040000 5131b4d4212b134d58fdaec4481a41239a69926c
e22212e758aa6a15e13c23ec8fb00f65bcf3a741 M lisp
I should have thought to check my other server (cyrus) - I can do so
when I get back to work next week. I'm happy to help troubleshoot but
need guidance on what to look for.

Yeah, this probably wasn't quite ready. There are a couple of other
problems with that patch that I'm looking at now, and hopefully I'll be
able to come up with something that addresses this as well, if I can
figure out what office365 is doing for encoding. If you have any hints
in that direction, let me know! Sorry for the breakage...

E

[Eric Abrahamsen]

Post by Dave Goldberg
All searches come up empty since this commit (per git-bisect)
046825b09ff6e23ec7dfacd451d10985615b9793 is the first bad commit
commit 046825b09ff6e23ec7dfacd451d10985615b9793
Date: Tue Jan 27 16:35:25 2015 +1100
* nnir.el (nnir-run-imap): Enable non-ASCII IMAP searches.
:040000 040000 5131b4d4212b134d58fdaec4481a41239a69926c
e22212e758aa6a15e13c23ec8fb00f65bcf3a741 M lisp
I should have thought to check my other server (cyrus) - I can do so
when I get back to work next week. I'm happy to help troubleshoot but
need guidance on what to look for.

The main problem here is that we're feeding a CHARSET argument to the
imap SEARCH command, but I really had no idea how to determine the
appropriate charset. Right now it's done using a value derived from the
`default-process-coding-system', which is bad because:

1. That variable doesn't exist in xemacs
2. It's probably just the wrong thing to do.

I guess what would be most helpful is if you could tell me your value
for default-process-coding-system, and then do a telnet session with
your exchange server. I'm not sure about exchange but, assuming you're
connecting with ssl, here's what I do with gmail:

openssl s_client -crlf -connect imap.gmail.com:993

Once you're in:

. login ***@address.com password
. select INBOX ; or whatever mailbox

(if hitting return doesn't seem to produce a response, try leaving out
the -crlf argument to openssl).

Then you should be able to search with:

. search charset UTF-8 subject "search string"

What I'd like to know is how the different charset arguments affect the
result. Find a string you know is in one of the headers, and try to
search with a few different charsets:

UTF-8
WINDOWS-1252
US-ASCII
ISO-8859-1
whatever you've got in 'default-process-coding-system'

And just see what comes up.

It might also be nice to see the CAPABILITY line that the server gives
you after you connect.

Sorry to give you all this homework! Until someone more knowledgeable
steps up, we're stuck with me muddling through :)

Eric

[Eric S Fraga]

Post by Eric Abrahamsen

Post by Dave Goldberg
All searches come up empty since this commit (per git-bisect)

This happens to me as well in latest gnus (as of yesterday, that
is). I've backed up to a version that works for now.

[...]

Post by Eric Abrahamsen
I guess what would be most helpful is if you could tell me your value
for default-process-coding-system, and then do a telnet session with
your exchange server. I'm not sure about exchange but, assuming you're

To give you a data point, I have followed your instructions for the IMAP
server I use which is outlook.office365.com (Microsoft :(). The only
charset that works is US-ASCII. This is rather annoying, to say the
least.

My default-process-coding-system is utf-8-unix.

HTH,
eric

[Eric Abrahamsen]

Post by Eric S Fraga

Post by Eric Abrahamsen

Post by Dave Goldberg
All searches come up empty since this commit (per git-bisect)

This happens to me as well in latest gnus (as of yesterday, that
is). I've backed up to a version that works for now.
[...]

Post by Eric Abrahamsen
I guess what would be most helpful is if you could tell me your value
for default-process-coding-system, and then do a telnet session with
your exchange server. I'm not sure about exchange but, assuming you're

To give you a data point, I have followed your instructions for the IMAP
server I use which is outlook.office365.com (Microsoft :(). The only
charset that works is US-ASCII. This is rather annoying, to say the
least.
My default-process-coding-system is utf-8-unix.

Oh man. That is helpful, though obviously also very depressing. I can't
believe they wouldn't let you input non-ascii search strings, though,
there must be some way around that...

More research! Probably this feature is currently too uncertain to leave
in trunk, it's going to mess with people's imap usage.

[e***@ucl.ac.uk]

On Tuesday, 3 Feb 2015 at 21:01, Eric Abrahamsen wrote:

[...]

Post by Eric Abrahamsen
More research! Probably this feature is currently too uncertain to leave
in trunk, it's going to mess with people's imap usage.

Yes, having this feature as it stands is not a good idea. I cannot use
gnus with it. Search is an integral part of my work flow. Please
revert these changes.

thanks,
eric

[Dave Goldberg]

Post by e***@ucl.ac.uk
[...]

Post by Eric Abrahamsen
More research! Probably this feature is currently too uncertain to leave
in trunk, it's going to mess with people's imap usage.

Yes, having this feature as it stands is not a good idea. I cannot use
gnus with it. Search is an integral part of my work flow. Please
revert these changes.

I had the exact same results with office365 as Eric did. No problem with searching the cyrus server so I didn't try the by-hand connection.

[Eric Abrahamsen]

Post by Dave Goldberg

Post by e***@ucl.ac.uk
[...]

Post by Eric Abrahamsen
More research! Probably this feature is currently too uncertain to leave
in trunk, it's going to mess with people's imap usage.

Yes, having this feature as it stands is not a good idea. I cannot use
gnus with it. Search is an integral part of my work flow. Please
revert these changes.

I had the exact same results with office365 as Eric did. No problem with searching the cyrus server so I didn't try the by-hand connection.

Thanks for checking! I've reverted the change.

[Eric S Fraga]

Post by Eric Abrahamsen
Thanks for checking! I've reverted the change.

Thanks. I've updated my gnus to latest from git and search works fine again.

[Katsumi Yamaoka]

Post by Eric Abrahamsen
1. That variable doesn't exist in xemacs

How about simply making it a user option (the default is utf-8)?
utf-8 is available in XEmacsen[1].

Post by Eric Abrahamsen
2. It's probably just the wrong thing to do.

I think so. The most reasonable solution might be to use
`default-buffer-file-coding-system' that is used for text
encoding, however the value of it is unreliable. I don't know
how, but on Cygwin `(set-language-environment "Japanese")' sets
it to `japanese-shift-jis-unix' even though the locale is set to
ja_JP.UTF-8, for instance. So is `default-process-coding-system'.

[1] XEmacs 21.4 users need to load Mule-UCS: (require 'un-define)

[Eric Abrahamsen]

Post by Katsumi Yamaoka

Post by Eric Abrahamsen
1. That variable doesn't exist in xemacs

How about simply making it a user option (the default is utf-8)?
utf-8 is available in XEmacsen[1].

Post by Eric Abrahamsen
2. It's probably just the wrong thing to do.

I think so. The most reasonable solution might be to use
`default-buffer-file-coding-system' that is used for text
encoding, however the value of it is unreliable. I don't know
how, but on Cygwin `(set-language-environment "Japanese")' sets
it to `japanese-shift-jis-unix' even though the locale is set to
ja_JP.UTF-8, for instance. So is `default-process-coding-system'.
[1] XEmacs 21.4 users need to load Mule-UCS: (require 'un-define)

There's also a variable `network-coding-system-alist' (nil on my
system). As far as I can tell, the "real" encoding system used for
talking to the imap server ought to be produced by a call that looks
like this:

(find-operation-coding-system 'open-network-stream name work-buffer host
service parameters)

Where the other arguments are produced inside of `open-protocol-stream'
aka `open-network-stream'. I tried eval'ing that while edebugging, and
it produced nil, which either means I did it wrong, or just that some
default value is being used instead -- perhaps just the buffer coding
system of the process buffer?

But even if we figure out a reliable way to find the right coding
system, there's still the issue of the imap servers and their various
coding quirks. I'm still surprised that searching for non-ascii terms is
such a difficult thing to do.

Eric

[Andreas Schwab]

Post by Eric Abrahamsen
(find-operation-coding-system 'open-network-stream name work-buffer host
service parameters)
Where the other arguments are produced inside of `open-protocol-stream'
aka `open-network-stream'. I tried eval'ing that while edebugging, and
it produced nil, which either means I did it wrong, or just that some
default value is being used instead -- perhaps just the buffer coding
system of the process buffer?

That just means network-coding-system-alist doesn't contain a match. It
doesn't try to find a definitive answer.

Andreas.

[Eric Abrahamsen]

Post by Andreas Schwab

Post by Eric Abrahamsen
(find-operation-coding-system 'open-network-stream name work-buffer host
service parameters)
Where the other arguments are produced inside of `open-protocol-stream'
aka `open-network-stream'. I tried eval'ing that while edebugging, and
it produced nil, which either means I did it wrong, or just that some
default value is being used instead -- perhaps just the buffer coding
system of the process buffer?

That just means network-coding-system-alist doesn't contain a match. It
doesn't try to find a definitive answer.

Yeah, that's what I figured. I might go ask on emacs.help.