nnir and notmuch?

[Rainer M Krug]

Now that I can search gmane (perfect - love it!) I can't search my local
imap server via notmuch anymore - any suggestions how I can achieve
both via nnir or "G G"?

I can obviously still run notmuch-search, and I also could binf=d it to
a new keyboard shortcut, but I would prefer that "G G" searches, when on
folders from the local maildir, using notmuch.

Is this possible?

Thanks,

Rainer

[Steinar Bang]

Post by Rainer M Krug
Now that I can search gmane (perfect - love it!) I can't search my local
imap server via notmuch anymore - any suggestions how I can achieve
both via nnir or "G G"?

One way would be to replace
(nnimap . imap)
in nnir-method-default-engines with
(nnimap . notmuch)

But that would mask native imap search for other IMAP servers you might
have.

[Rasmus]

Post by Steinar Bang

Post by Rainer M Krug
Now that I can search gmane (perfect - love it!) I can't search my local
imap server via notmuch anymore - any suggestions how I can achieve
both via nnir or "G G"?

One way would be to replace
(nnimap . imap)
in nnir-method-default-engines with
(nnimap . notmuch)
But that would mask native imap search for other IMAP servers you might
have.

If Rainer is using nnmaildir it might be less of an issue.

A less elegant way would be to replace G-G. Something like:

(defun my-gnus-G-G (NNIR-EXTRA-PARMS &optional SPECS)
"Chose the right search"
(interactive "P")
(let ((backend (car-safe
(split-string (or (gnus-group-group-name) "") ":"))))
(unless (eq "" backend)
(case backend
("nntp+gmane"
(gnus-group-make-nnir-group NNIR-EXTRA-PARMS SPECS))
("..." (search-using-notmuch))
(otherwise
(gnus-group-make-nnir-group NNIR-EXTRA-PARMS SPECS))))))

(define-key gnus-group-mode-map (kbd "G G") 'my-gnus-G-G)

I don't know how well it would work in practice.

—Rasmus

[Rainer M Krug]

Post by Rasmus

Post by Steinar Bang

Post by Rainer M Krug
Now that I can search gmane (perfect - love it!) I can't search my local
imap server via notmuch anymore - any suggestions how I can achieve
both via nnir or "G G"?

One way would be to replace
(nnimap . imap)
in nnir-method-default-engines with
(nnimap . notmuch)
But that would mask native imap search for other IMAP servers you might
have.

If Rainer is using nnmaildir it might be less of an issue.

I am using local imap, but only local imap - I want to leave the maildir
under the control of dovecot.
That would most likely work - but as you say less elegant and I would
like to keep to much hacking out of my setup. But I will keep it in
mind and use it if everything else fails.

Rainer

Post by Rasmus
(defun my-gnus-G-G (NNIR-EXTRA-PARMS &optional SPECS)
"Chose the right search"
(interactive "P")
(let ((backend (car-safe
(split-string (or (gnus-group-group-name) "") ":"))))
(unless (eq "" backend)
(case backend
("nntp+gmane"
(gnus-group-make-nnir-group NNIR-EXTRA-PARMS SPECS))
("..." (search-using-notmuch))
(otherwise
(gnus-group-make-nnir-group NNIR-EXTRA-PARMS SPECS))))))
(define-key gnus-group-mode-map (kbd "G G") 'my-gnus-G-G)
I don't know how well it would work in practice.
—Rasmus

[Rainer M Krug]

<#secure method=pgpmime mode=sign>

Post by Steinar Bang

Post by Rainer M Krug
Now that I can search gmane (perfect - love it!) I can't search my
local imap server via notmuch anymore - any suggestions how I can
achieve both via nnir or "G G"?

One way would be to replace
(nnimap . imap) in nnir-method-default-engines with
(nnimap . notmuch)

I had

,----
| '(nnimap "Maildir"
| (nnimap-stream shell) (nnimap-shell-program
| "/usr/local/opt/dovecot/libexec/dovecot/imap -o
| mail_location=maildir:$HOME/Maildir")
| (nnir-search-engine 'notmuch)))
`----

in my gnus file, but it produced an error message.
I am just trying your suggestion, and I get the following:

,----
| notmuch args: search --format=text --output=files hallo
| Massaging notmuch output...done
| Search produced empty results.
| gnus-group-read-ephemeral-group: Couldn't request group: Search produced empty results.
`----

although the search on the commandline

,----
| notmuch search --format=text --output=files Rainer
`----

yields many results.

So - when setting nnir-method-default-engines, nnir uses notmuch but
does not return any results, while using nnir-search-engine does not
work.

Any suggestions from here? I am by the way using Gnus v5.13 - is notmuch
integrated as an nnir search engine?

Post by Steinar Bang
But that would mask native imap search for other IMAP servers you
might have.

No problem for me - I am using offlineimap to get my mail and then only
access the local imap server and no other - at least at the moment.

Cheers,

Rainer
--
Rainer M. Krug
email: Rainer<at>krugs<dot>de
PGP: 0x0F52F982

[Steinar Bang]

Post by Rainer M Krug
Any suggestions from here?

Sorry, no. I just read the nnir.el to see how nnir backends are picked.

[Andrew Cohen]

Post by Rainer M Krug
Now that I can search gmane (perfect - love it!) I can't search
my local imap server via notmuch anymore - any suggestions how I
can achieve both via nnir or "G G"?

Rainer> I had

Rainer> ,---- | '(nnimap "Maildir" | (nnimap-stream shell)
Rainer> (nnimap-shell-program |
Rainer> "/usr/local/opt/dovecot/libexec/dovecot/imap -o |
Rainer> mail_location=maildir:$HOME/Maildir") | (nnir-search-engine
Rainer> 'notmuch))) `----

Rainer> in my gnus file, but it produced an error message.


"G G" should use whatever search engine you have configured. Setting
nnir-search-engine in the server spec to notmuch will make "G G" use
notmuch for that server. You don't say what error you received; I think
that you should use the name of the server rather than a symbol (that
is, try (nnir-search-engine notmuch) without the apostrophe).

You will have to configure variables for notmuch. I don't use the
notmuch engine so I can't help you with this configuration, but you can
read the notmuch docs and maybe someone on the list who does use it can
chime in.

Regards,
Andy

[Rainer M Krug]

Post by Andrew Cohen

Post by Rainer M Krug
Now that I can search gmane (perfect - love it!) I can't search
my local imap server via notmuch anymore - any suggestions how I
can achieve both via nnir or "G G"?

Rainer> I had
Rainer> ,---- | '(nnimap "Maildir" | (nnimap-stream shell)
Rainer> (nnimap-shell-program |
Rainer> "/usr/local/opt/dovecot/libexec/dovecot/imap -o |
Rainer> mail_location=maildir:$HOME/Maildir") | (nnir-search-engine
Rainer> 'notmuch))) `----
Rainer> in my gnus file, but it produced an error message.
"G G" should use whatever search engine you have configured. Setting
nnir-search-engine in the server spec to notmuch will make "G G" use
notmuch for that server. You don't say what error you received; I think
that you should use the name of the server rather than a symbol (that
is, try (nnir-search-engine notmuch) without the apostrophe).

Sounds like an idea - I'll try it out. But I do not assume that that is
working if it is not when notmuch is set as default search engine for
imap (I use only local imap with maildir as backend, which is indexed by
notuch).

Post by Andrew Cohen
You will have to configure variables for notmuch.

Notmuch from the command line is working - if you refer to parameters
relating to nnir, which ones do I have to set?

Post by Andrew Cohen
I don't use the notmuch engine so I can't help you with this
configuration, but you can read the notmuch docs and maybe someone on
the list who does use it can chime in.

Thanks,

Rainer

Post by Andrew Cohen
Regards,
Andy

[Eric Abrahamsen]

Post by Rainer M Krug
<#secure method=pgpmime mode=sign>

Post by Steinar Bang

Post by Rainer M Krug
Now that I can search gmane (perfect - love it!) I can't search my
local imap server via notmuch anymore - any suggestions how I can
achieve both via nnir or "G G"?

One way would be to replace
(nnimap . imap) in nnir-method-default-engines with
(nnimap . notmuch)

I had
,----
| '(nnimap "Maildir"
| (nnimap-stream shell) (nnimap-shell-program
| "/usr/local/opt/dovecot/libexec/dovecot/imap -o
| mail_location=maildir:$HOME/Maildir")
| (nnir-search-engine 'notmuch)))
`----
in my gnus file, but it produced an error message.
,----
| notmuch args: search --format=text --output=files hallo
| Massaging notmuch output...done
| Search produced empty results.
| gnus-group-read-ephemeral-group: Couldn't request group: Search produced empty results.
`----

For a while I ran notmuch and imap configured like this, and got a lot
of the same problems, because I was a bit confused about how all the
bits and pieces interacted. I think the problem is that notmuch returns
its hits based on filesystem paths. These paths don't necessarily match
up with how imap locates its messages, though -- things like mailboxes
transformations and namespaces can mean that imap sees the location of a
message in a way that doesn't jive with the filesystem path.

If you've got multiple Gnus backends in effect, another awkwardness is
that notmuch will probably return all the hits from all backends,
whereas nnir will attempt to filter out anything that doesn't belong to
the group/server from which you initiated the search.

I don't know all the grisly details, or whether it could be made to
work: when I realized what was happening I just stopped using notmuch as
a nnir engine for nnimap messages. Now I have a Frankenstein search
setup, using notmuch-search by itself, and the imap nnir engine, in
parallel. I'm shifting over to using nnir only, and will eventually get
rid of notmuch.

Not very helpful, I know, but I think there are fundamental
incompatibilities in specifying notmuch as a nnir engine for imap
servers.

Eric

[Rainer M Krug]

Post by Eric Abrahamsen

Post by Rainer M Krug
<#secure method=pgpmime mode=sign>

Post by Steinar Bang

Post by Rainer M Krug
Now that I can search gmane (perfect - love it!) I can't search my
local imap server via notmuch anymore - any suggestions how I can
achieve both via nnir or "G G"?

One way would be to replace
(nnimap . imap) in nnir-method-default-engines with
(nnimap . notmuch)

I had
,----
| '(nnimap "Maildir"
| (nnimap-stream shell) (nnimap-shell-program
| "/usr/local/opt/dovecot/libexec/dovecot/imap -o
| mail_location=maildir:$HOME/Maildir")
| (nnir-search-engine 'notmuch)))
`----
in my gnus file, but it produced an error message.
,----
| notmuch args: search --format=text --output=files hallo
| Massaging notmuch output...done
| Search produced empty results.
| gnus-group-read-ephemeral-group: Couldn't request group: Search produced empty results.
`----

For a while I ran notmuch and imap configured like this, and got a lot
of the same problems, because I was a bit confused about how all the
bits and pieces interacted. I think the problem is that notmuch returns
its hits based on filesystem paths. These paths don't necessarily match
up with how imap locates its messages, though -- things like mailboxes
transformations and namespaces can mean that imap sees the location of a
message in a way that doesn't jive with the filesystem path.

This might be the reason - makes sense. As I want to use dovecot to
access my email stored in maildir (for notmuch, as this is not modifying
thew mails when only used for searching), I think I can not use it this
way. Makes perfect sense.

Post by Eric Abrahamsen
If you've got multiple Gnus backends in effect, another awkwardness is
that notmuch will probably return all the hits from all backends,
whereas nnir will attempt to filter out anything that doesn't belong to
the group/server from which you initiated the search.

I have only the local imap and gmane.

Post by Eric Abrahamsen
I don't know all the grisly details, or whether it could be made to
work: when I realized what was happening I just stopped using notmuch
as a nnir engine for nnimap messages.

I think I will do the same here.

Post by Eric Abrahamsen
Now I have a Frankenstein search setup, using notmuch-search by
itself, and the imap nnir engine, in parallel.

Sounds like the best solution - and then just bind notmuch-search to a
keyboard shortcut.

Post by Eric Abrahamsen
I'm shifting over to using nnir only, and will eventually get rid of
notmuch.

I tried the normal search on y imap folders, and it is incredibly slow
(folders with 10 000 and more emails). Also, my archive is split into
years, and a topic can span two years - so the normal imap search is not
an option for me here.

Post by Eric Abrahamsen
Not very helpful, I know, but I think there are fundamental
incompatibilities in specifying notmuch as a nnir engine for imap
servers.

In contrast - quite helpful, as I understood why it is not working to
use notmuch for an imap backend. And as my setup is running nicely, I do
not intend to change it.

Thanks a lot,

Rainer

Post by Eric Abrahamsen
Eric

[Eric Abrahamsen]

Post by Rainer M Krug

Post by Eric Abrahamsen

Post by Rainer M Krug
<#secure method=pgpmime mode=sign>

Post by Steinar Bang

Post by Rainer M Krug
Now that I can search gmane (perfect - love it!) I can't search my
local imap server via notmuch anymore - any suggestions how I can
achieve both via nnir or "G G"?

One way would be to replace
(nnimap . imap) in nnir-method-default-engines with
(nnimap . notmuch)

I had
,----
| '(nnimap "Maildir"
| (nnimap-stream shell) (nnimap-shell-program
| "/usr/local/opt/dovecot/libexec/dovecot/imap -o
| mail_location=maildir:$HOME/Maildir")
| (nnir-search-engine 'notmuch)))
`----
in my gnus file, but it produced an error message.
,----
| notmuch args: search --format=text --output=files hallo
| Massaging notmuch output...done
| Search produced empty results.
| gnus-group-read-ephemeral-group: Couldn't request group: Search produced empty results.
`----

For a while I ran notmuch and imap configured like this, and got a lot
of the same problems, because I was a bit confused about how all the
bits and pieces interacted. I think the problem is that notmuch returns
its hits based on filesystem paths. These paths don't necessarily match
up with how imap locates its messages, though -- things like mailboxes
transformations and namespaces can mean that imap sees the location of a
message in a way that doesn't jive with the filesystem path.

This might be the reason - makes sense. As I want to use dovecot to
access my email stored in maildir (for notmuch, as this is not modifying
thew mails when only used for searching), I think I can not use it this
way. Makes perfect sense.

Post by Eric Abrahamsen
If you've got multiple Gnus backends in effect, another awkwardness is
that notmuch will probably return all the hits from all backends,
whereas nnir will attempt to filter out anything that doesn't belong to
the group/server from which you initiated the search.

I have only the local imap and gmane.

Post by Eric Abrahamsen
I don't know all the grisly details, or whether it could be made to
work: when I realized what was happening I just stopped using notmuch
as a nnir engine for nnimap messages.

I think I will do the same here.

Post by Eric Abrahamsen
Now I have a Frankenstein search setup, using notmuch-search by
itself, and the imap nnir engine, in parallel.

Sounds like the best solution - and then just bind notmuch-search to a
keyboard shortcut.

Post by Eric Abrahamsen
I'm shifting over to using nnir only, and will eventually get rid of
notmuch.

I tried the normal search on y imap folders, and it is incredibly slow
(folders with 10 000 and more emails). Also, my archive is split into
years, and a topic can span two years - so the normal imap search is not
an option for me here.

You can search more than one group with nnir, by marking the desired
groups, or calling it on a topic heading, or on a whole server in the
*Server* buffer.

I just set up a new system with the lucene text indexer, and it works
much better. I have a blog post mostly written about this, and can post
it sooner if you think it might be helpful.

Post by Rainer M Krug

Post by Eric Abrahamsen
Not very helpful, I know, but I think there are fundamental
incompatibilities in specifying notmuch as a nnir engine for imap
servers.

In contrast - quite helpful, as I understood why it is not working to
use notmuch for an imap backend. And as my setup is running nicely, I do
not intend to change it.
Thanks a lot,
Rainer

Post by Eric Abrahamsen
Eric

[Adam Sjøgren]

Post by Eric Abrahamsen
I just set up a new system with the lucene text indexer, and it works
much better.

Than what?


Just curious,

Adam

[Eric Abrahamsen]

Post by Adam Sjøgren

Post by Eric Abrahamsen
I just set up a new system with the lucene text indexer, and it works
much better.

Than what?

Than the native imap search, which is god-awfully slow.

To be clear, I use dovecot as a local imap cache. Originally I wasn't
using it as a daemon: mail came in via isync, which called dovecot at
/usr/lib/dovecot/imap. Then gnus called the same executable via
nnimap-shell-program to get the mail. Dovecot only "woke up" when one or
the other program called it.

Nnir search just used whatever search facilities dovecot provided out of
the box. It was slow. I read about integrating lucene into dovecot, and
as far as I could tell it wouldn't work when you were just calling
dovecot's imap executable -- you had to be running a dovecot instance.

So I switched to an always-running dovecot instance, configured the
lucene indexer, and now the exact same nnir searches go like lightning.

That's basically a summary of the blog post :)

E

[Alan Schmitt]

Post by Eric Abrahamsen
To be clear, I use dovecot as a local imap cache. Originally I wasn't
using it as a daemon: mail came in via isync, which called dovecot at
/usr/lib/dovecot/imap. Then gnus called the same executable via
nnimap-shell-program to get the mail. Dovecot only "woke up" when one or
the other program called it.
Nnir search just used whatever search facilities dovecot provided out of
the box. It was slow. I read about integrating lucene into dovecot, and
as far as I could tell it wouldn't work when you were just calling
dovecot's imap executable -- you had to be running a dovecot instance.
So I switched to an always-running dovecot instance, configured the
lucene indexer, and now the exact same nnir searches go like lightning.
That's basically a summary of the blog post :)

I'd be really curious to hear more about this. Please let us know when
you publish your post.

Alan

[Eric Abrahamsen]

Post by Alan Schmitt

Post by Eric Abrahamsen
To be clear, I use dovecot as a local imap cache. Originally I wasn't
using it as a daemon: mail came in via isync, which called dovecot at
/usr/lib/dovecot/imap. Then gnus called the same executable via
nnimap-shell-program to get the mail. Dovecot only "woke up" when one or
the other program called it.
Nnir search just used whatever search facilities dovecot provided out of
the box. It was slow. I read about integrating lucene into dovecot, and
as far as I could tell it wouldn't work when you were just calling
dovecot's imap executable -- you had to be running a dovecot instance.
So I switched to an always-running dovecot instance, configured the
lucene indexer, and now the exact same nnir searches go like lightning.
That's basically a summary of the blog post :)

I'd be really curious to hear more about this. Please let us know when
you publish your post.
Alan

Will do!

[Adam Sjøgren]

Post by Eric Abrahamsen

Post by Adam Sjøgren

Post by Eric Abrahamsen
much better.

Than what?

Than the native imap search, which is god-awfully slow.

Ah, cool - thanks for the explanation.


:-),

Adam

[Eric S Fraga]

Eric et al.,

All the proposed solutions require local imap mirroring if I understand
correctly. This is an appealing approach for a number of
reasons. However, I have been spectacularly unsuccessful in getting
either offlineimap or isync to synchronise with an MS Exchange
server. I get filename errors or the transfer simply quits without ever
managing to complete the first synchronisation. (I should nte that I
haven't tried doing this in over a year now... maybe things have
improved?) Has anybody here managed it?

Thanks,
eric

[Rainer M Krug]

Post by Eric S Fraga
Eric et al.,

Hi Eric,

Post by Eric S Fraga
All the proposed solutions require local imap mirroring if I understand
correctly. This is an appealing approach for a number of
reasons. However, I have been spectacularly unsuccessful in getting
either offlineimap or isync to synchronise with an MS Exchange
server. I get filename errors or the transfer simply quits without ever
managing to complete the first synchronisation. (I should nte that I
haven't tried doing this in over a year now... maybe things have
improved?) Has anybody here managed it?

I use offlineimap it for gmail (works), iCloud (works) and a university
accout where they are running ... Exchange server.

Below is the relevant part of my .offlineimaprc file.

My offlineimap synchs to a local dovecot server (lucerne indexing
working!) and puts all folders from this account into a folder named
SUN.

I do the same for my other accounts, so I easily know which ones are
From which account.

But my guess is that the configuration of the Exchange server is probably
different - but if you use imap access at the moment, it should work.

Cheers,

Rainer

--8<---------------cut here---------------start------------->8---
## %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
## %% SUN %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
## %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

[Repository LocalSUN]
type = IMAP
remotehost = localhost
remoteuser = MYUSERNAME
remotepasseval = get_keychain_pass(account="MYUSERNAME")
holdconnectionopen = yes
maxconnections = 3
nametrans = lambda folder: re.sub('^SUN.', '', folder)
folderfilter = lambda folder: folder.startswith('SUN')
keepalive = 60
holdconnectionopen = yes

[Repository RemoteSUN]
type = IMAP
remotehost = imap.sun.ac.za
# port = 143
remoteuser = MYUSERNAME
remotepasseval = get_keychain_pass(account="MYUSERNAME")
######
maxconnections = 1
nametrans = lambda foldername: 'SUN.' + foldername
######
folderfilter = lambda folder: folder in ['INBOX']
## For now:
# readonly = true
# createfolders = false
######
keepalive = 60
holdconnectionopen = yes

[Account SUN]
localrepository = LocalSUN
remoterepository = RemoteSUN
######
status_backend = sqlite
######
# postsynchook = ionice -c 3 chrt --idle 0 /bin/sh -c "notmuch new"
# postsynchook = ionice -c 3 chrt --idle 0 /bin/sh -c "notmuch new && afew --tag --verbose --new"
######
######
## For daemon mode
# Minutes between syncs
autorefresh = 2
# Number of quick-syncs between autorefreshes. Quick-syncs do not update if the
# only changes were to IMAP flags
quick = 10
--8<---------------cut here---------------end--------------->8---

Post by Eric S Fraga
Thanks,
eric

[Eric S Fraga]

Post by Rainer M Krug
I use offlineimap it for gmail (works), iCloud (works) and a university
accout where they are running ... Exchange server.
Below is the relevant part of my .offlineimaprc file.

Thanks. My configuration was not radically different other than I had
to exclude a large number of folders (ones like Calendar which are not
real email folders) but I'll play around some more when I have a
chance.

[Adam Sjøgren]

I have been spectacularly unsuccessful in getting either offlineimap
or isync to synchronise with an MS Exchange server. I get filename
errors or the transfer simply quits without ever managing to complete
the first synchronisation. (I should nte that I haven't tried doing
this in over a year now... maybe things have improved?) Has anybody
here managed it?

My workaround is to use Exchange IMAP as a glorified POP3, so I just get
new emails from Exchange, but never manipulate anything on Exchange.

I.e. as described in:

* http://gnus.org/manual/gnus_179.html#SEC179


Mvh.

Adam

[Eric S Fraga]

On Friday, 26 Sep 2014 at 11:54, Adam Sjøgren wrote:

[...]

Post by Adam Sjøgren
My workaround is to use Exchange IMAP as a glorified POP3, so I just get
new emails from Exchange, but never manipulate anything on Exchange.

Ummm, intriguing. Sort of /back to the future/. I need to think about
this approach as I am definitely used to having IMAP access to my email
from a number of devices...

thanks,
eric