blog post on gnus, dovecot, and lucene

[Eric Abrahamsen]

Hi,

I mentioned a bit ago that I had shifted my gnus/dovecot setup, and
would blog about it. I finally did that:

http://ericabrahamsen.net/tech/2014/oct/gnus-dovecot-lucene.html

The gist of it is how to move to an always-running dovecot daemon, fed
by isync, and incorporating lucene searches into that.

Enjoy!

Eric

[Igor Sosa Mayor]

Post by Eric Abrahamsen
Hi,
I mentioned a bit ago that I had shifted my gnus/dovecot setup, and
http://ericabrahamsen.net/tech/2014/oct/gnus-dovecot-lucene.html
The gist of it is how to move to an always-running dovecot daemon, fed
by isync, and incorporating lucene searches into that.

interesting! Thanks!

[Feng Shu]

Post by Igor Sosa Mayor

Post by Eric Abrahamsen
Hi,
I mentioned a bit ago that I had shifted my gnus/dovecot setup, and
http://ericabrahamsen.net/tech/2014/oct/gnus-dovecot-lucene.html>
The gist of it is how to move to an always-running dovecot daemon, fed
by isync, and incorporating lucene searches into that.

I use gnus+dovecot+offlineimap+leafnod+rss2email

This is my configure: [[https://github.com/tumashu/emacs-helper]]

Post by Igor Sosa Mayor
interesting! Thanks!

--

[Rasmus]

Hi,

Post by Feng Shu

Post by Eric Abrahamsen
I mentioned a bit ago that I had shifted my gnus/dovecot setup, and
http://ericabrahamsen.net/tech/2014/oct/gnus-dovecot-lucene.html>
The gist of it is how to move to an always-running dovecot daemon, fed
by isync, and incorporating lucene searches into that.

I use gnus+dovecot+offlineimap+leafnod+rss2email
This is my configure: [[https://github.com/tumashu/emacs-helper]]

Thanks for sharing Eric and Feng. I personally run Dovecot the way
Vincent suggests.

The one thing I'm lacking is proper IMAP IDLE support to automatically
download new emails (not as a chronjob). The one built into
offlineimap is okay-ish, but doesn't always see all changes so I still
have to run it manually. I'm not sure how reliable it is when the
network changes or when I suspend my machine.

Does any of you have a reliable IDLE solution?

Cheers,
Rasmus

[Vincent Bernat]

Post by Eric Abrahamsen
I mentioned a bit ago that I had shifted my gnus/dovecot setup, and
http://ericabrahamsen.net/tech/2014/oct/gnus-dovecot-lucene.html
The gist of it is how to move to an always-running dovecot daemon, fed
by isync, and incorporating lucene searches into that.

Note that you don't have to run dovecot as a server. You can run it as a
process. In mbsyncrc:

Tunnel "/usr/lib/dovecot/imap -o mail_location=maildir:~/.mbsync/mails/XXXXX"

In Gnus:

#+BEGIN_SRC elisp
(nnimap-stream shell)
(nnimap-shell-program "/usr/lib/dovecot/imap -o mail_location=maildir:~/.mbsync/mails/XXXXX")
#+END_SRC

[Eric Abrahamsen]

Post by Vincent Bernat

Post by Eric Abrahamsen
I mentioned a bit ago that I had shifted my gnus/dovecot setup, and
http://ericabrahamsen.net/tech/2014/oct/gnus-dovecot-lucene.html
The gist of it is how to move to an always-running dovecot daemon, fed
by isync, and incorporating lucene searches into that.

Note that you don't have to run dovecot as a server. You can run it as a
Tunnel "/usr/lib/dovecot/imap -o mail_location=maildir:~/.mbsync/mails/XXXXX"
#+BEGIN_SRC elisp
(nnimap-stream shell)
(nnimap-shell-program "/usr/lib/dovecot/imap -o mail_location=maildir:~/.mbsync/mails/XXXXX")
#+END_SRC

Yes, hence the section titled "The Problem" in that blog post :) Perhaps
I could have made that clearer!

So far as I can tell, if you're calling dovecot as a process, you can't
integrate lucene text indexing -- that only works when running dovecot
as a daemon.

In fact that was the whole impetus for the shift, and the blog post:
going from dovecot-the-process to dovecot-the-daemon, because of the
need for search indexing.

E

[Vincent Bernat]

Post by Eric Abrahamsen

Post by Vincent Bernat
Note that you don't have to run dovecot as a server. You can run it as a
Tunnel "/usr/lib/dovecot/imap -o mail_location=maildir:~/.mbsync/mails/XXXXX"
#+BEGIN_SRC elisp
(nnimap-stream shell)
(nnimap-shell-program "/usr/lib/dovecot/imap -o mail_location=maildir:~/.mbsync/mails/XXXXX")
#+END_SRC

Yes, hence the section titled "The Problem" in that blog post :) Perhaps
I could have made that clearer!

Oh, it was pretty clear but I didn't read from the beginning because I
thought this was a post about reading mails offline with full text
search available! ;-)

Post by Eric Abrahamsen
So far as I can tell, if you're calling dovecot as a process, you can't
integrate lucene text indexing -- that only works when running dovecot
as a daemon.
going from dovecot-the-process to dovecot-the-daemon, because of the
need for search indexing.

I didn't notice that. I added that to my dovecot.conf:

plugin {
fts = lucene
fts_lucene = whitespace_chars=@.
}

And the search became faster. However, I usually don't use FTS (only
search on recipient or title). If I check right now, I notice that the
indexes are not up-to-date. So, you may be right.

[Eric Abrahamsen]

Post by Vincent Bernat

Post by Eric Abrahamsen

Post by Vincent Bernat
Note that you don't have to run dovecot as a server. You can run it as a
Tunnel "/usr/lib/dovecot/imap -o mail_location=maildir:~/.mbsync/mails/XXXXX"
#+BEGIN_SRC elisp
(nnimap-stream shell)
(nnimap-shell-program "/usr/lib/dovecot/imap -o mail_location=maildir:~/.mbsync/mails/XXXXX")
#+END_SRC

Yes, hence the section titled "The Problem" in that blog post :) Perhaps
I could have made that clearer!

Oh, it was pretty clear but I didn't read from the beginning because I
thought this was a post about reading mails offline with full text
search available! ;-)

Post by Eric Abrahamsen
So far as I can tell, if you're calling dovecot as a process, you can't
integrate lucene text indexing -- that only works when running dovecot
as a daemon.
going from dovecot-the-process to dovecot-the-daemon, because of the
need for search indexing.

plugin {
fts = lucene
}
And the search became faster. However, I usually don't use FTS (only
search on recipient or title). If I check right now, I notice that the
indexes are not up-to-date. So, you may be right.

But it's pretty interesting that that did something -- I wasn't sure
that calling dovecot as a process would even invoke the configuration
files at all. But you don't think the indexes were updated, huh?

It might be worth amending the blog post at some point. I've asked about
this stuff on the dovecot mailing list, but no one seemed to know.

E

[Vincent Bernat]

Post by Eric Abrahamsen

Post by Vincent Bernat
plugin {
fts = lucene
}
And the search became faster. However, I usually don't use FTS (only
search on recipient or title). If I check right now, I notice that the
indexes are not up-to-date. So, you may be right.

But it's pretty interesting that that did something -- I wasn't sure
that calling dovecot as a process would even invoke the configuration
files at all. But you don't think the indexes were updated, huh?
It might be worth amending the blog post at some point. I've asked about
this stuff on the dovecot mailing list, but no one seemed to know.

I have done a doveadm fts rescan to get those indexes updated. So, maybe
the solution could be a cronjob?

[Eric Abrahamsen]

Post by Vincent Bernat

Post by Eric Abrahamsen

Post by Vincent Bernat
plugin {
fts = lucene
}
And the search became faster. However, I usually don't use FTS (only
search on recipient or title). If I check right now, I notice that the
indexes are not up-to-date. So, you may be right.

But it's pretty interesting that that did something -- I wasn't sure
that calling dovecot as a process would even invoke the configuration
files at all. But you don't think the indexes were updated, huh?
It might be worth amending the blog post at some point. I've asked about
this stuff on the dovecot mailing list, but no one seemed to know.

I have done a doveadm fts rescan to get those indexes updated. So, maybe
the solution could be a cronjob?

Right, that should work fine. At that point, though, I'm inclined just
to have dovecot run as a daemon...

[Rainer M Krug]

Post by Eric Abrahamsen

Post by Vincent Bernat

Post by Eric Abrahamsen

Post by Vincent Bernat
plugin {
fts = lucene
}
And the search became faster. However, I usually don't use FTS (only
search on recipient or title). If I check right now, I notice that the
indexes are not up-to-date. So, you may be right.

But it's pretty interesting that that did something -- I wasn't sure
that calling dovecot as a process would even invoke the configuration
files at all. But you don't think the indexes were updated, huh?
It might be worth amending the blog post at some point. I've asked about
this stuff on the dovecot mailing list, but no one seemed to know.

I have done a doveadm fts rescan to get those indexes updated. So, maybe
the solution could be a cronjob?

Instead of a cronjob, one could run it after the sync with offlinesync
(I am sure mbsync can do it as well) as a hook which is executed
automatically. But there is still the issue of updating after moving
messages from gnus.

Post by Eric Abrahamsen
Right, that should work fine. At that point, though, I'm inclined just
to have dovecot run as a daemon...

I think the daemon is the better way to do it.

Cheers,

Rainer

[Eric Abrahamsen]

Post by Eric Abrahamsen
Hi,
I mentioned a bit ago that I had shifted my gnus/dovecot setup, and
http://ericabrahamsen.net/tech/2014/oct/gnus-dovecot-lucene.html
The gist of it is how to move to an always-running dovecot daemon, fed
by isync, and incorporating lucene searches into that.

Nice - thanks.
The idea of using virtual users under dovecot is intriguing - it makes
all the name translations I am doing in offlineimap to keep my accounts
in different folder structures unneccessary. Could you please elaborrate
a little bit on your setup in this regard and where the emails are
actually stored? I might be implementing th same approach because it
sounds so simple!

I can't claim to understand the entire configuration procedure, as I was
just diving "deep enough" into dovecot, but the trick is this bit from
the dovecot.conf file:

passdb {
driver = passwd-file
args = /etc/dovecot/passwd
}

userdb {
driver = static
args = uid=eric gid=users home=/home/eric/.mail/%d/%n
default_fields = mail=maildir:/home/eric/.mail/%d/%n/mail
}


Particularly the userdb section. I understand this to be saying: "each
virtual user should be run as uid "eric" and gid "users", and each user
will have their home directory under /home/eric/.mail, in a directory
that looks like domainname/username. Furthermore, the mail for each user
will be stored in maildir format, in a mail/ subdirectory in that user's
home directory." It's like making viritual home directories for each
virtual user.

So now my directory structure looks like:

~/.mail/ericabrahamsen.net/
└── eric
└── mail
├── cur
├── dovecot.index
[ more dovecot stuff]
├── lucene-indexes
├── new
├── subscriptions
└── tmp
~/.mail/paper-republic.org/
├── eric
│ └── mail
│ ├── cur
│ ├── dovecot.index
│ [ etc ]
│ ├── lucene-indexes
│ ├── new
│ ├── subscriptions
│ └── tmp
└── info
└── mail
├── cur
├── dovecot.index
[ more dovecot stuff]
├── lucene-indexes
├── new
├── subscriptions
└── tmp

So in cases where I have more than one user at the same domain name,
both users are nested under the domain name directory.

In hindsight the extra mail/ directory was probably unnecessary, I had
some idea that other files might go above that mail/ directory.

Hope that helps,
Eric