Trying to prune the registry because it's full; apply: registry max-size limit reached

Discussion:

Tassilo Horn

2015-03-30 06:12:20 UTC

Hi all,

just now when I wanted to enter this group, I got the error message

Trying to prune the registry because it's full
apply: registry max-size limit reached

with this backtrace:

--8<---------------cut here---------------start------------->8---
Debugger entered: ((cl-assertion-failed (not (registry-full db)) "registry max-size limit reached"))
cl--assertion-failed((not (registry-full db)) "registry max-size limit reached" nil nil)
registry-insert([eieio-class-tag--registry-db "~/.gnus.d/.gnus.registry.eieio" 0.2 10000 0.1...])
gnus-registry-insert( "<59afc63a-24c7-4b7b-89d8-***@googlegroups.com>" ((creation-time (21784 58735 12080 293000)) (group) (sender) (subject)))
gnus-registry-get-or-make-entry("<59afc63a-24c7-4b7b-89d8-***@googlegroups.com>")
gnus-registry-get-id-key("<59afc63a-24c7-4b7b-89d8-***@googlegroups.com>" group)
gnus-registry-register-message-ids()
run-hooks(gnus-summary-prepare-hook)
apply(run-hooks gnus-summary-prepare-hook)
gnus-run-hooks(gnus-summary-prepare-hook)
gnus-summary-prepare()
gnus-summary-read-group-1("nnimap+Fastmail:INBOX.mailinglists.clojure" nil t nil nil nil)
gnus-summary-read-group("nnimap+Fastmail:INBOX.mailinglists.clojure" nil t nil nil nil nil)
gnus-group-read-group(nil t)
gnus-group-select-group(nil)
gnus-topic-select-group(nil)
funcall-interactively(gnus-topic-select-group nil)
call-interactively(gnus-topic-select-group nil nil)
command-execute(gnus-topic-select-group)
--8<---------------cut here---------------end--------------->8---

My registry-related settings are just

--8<---------------cut here---------------start------------->8---
(setq gnus-registry-ignored-groups '(("^nntp" t) ("^nndraft" t)
("^nnir" t)
("training" t) ("Junk" t)
("Trash" t) ("Spam" t))
;; Don't track anything except for the message ids.
gnus-registry-track-extra nil
gnus-registry-max-entries 10000)

(gnus-registry-initialize)
--8<---------------cut here---------------end--------------->8---

I had thought that with a maximum number of entries, as soon as I hit
it, the registry would prune the oldest entries to regain some space.
But according to the messages, it seems that pruning didn't succeed.

For the time being, I have increased `gnus-registry-max-entries'. Is
that a bug? I mean, given that `gnus-registry-prune-factor' is on its
standard value of 0.1, pruning should have left me with a registry of
9000 entries and then insertion should have worked again, according to
the docs:

,----[ (info "(gnus)Gnus Registry Setup") ]
| -- Variable: gnus-registry-max-entries
| The number (an integer or ‘nil’ for unlimited) of entries the
| registry will keep. If the registry has reached or exceeded this
| size, it will reject insertion of new entries.
|
| -- Variable: gnus-registry-prune-factor
| This option (a float between 0 and 1) controls how much the
| registry is cut back during pruning. In order to prevent constant
| pruning, the registry will be pruned back to less than
| ‘gnus-registry-max-entries’. This option controls exactly how much
| less: the target is calculated as the maximum number of entries
| minus the maximum number times this factor. The default is 0.1:
| i.e., if your registry is limited to 50000 entries, pruning will
| try to cut back to 45000 entries. Entries with keys marked as
| precious will not be pruned.
`----

Hm, actually the docs don't really mention that pruning will take place
automatically in case I'm hitting the limit but according to the
message I got before the error, I think it does (and everything else
wouldn't make sense anyway).

Bye,
Tassilo

Eric Abrahamsen

2015-03-30 06:57:35 UTC

Permalink

Post by Tassilo Horn
Hi all,
just now when I wanted to enter this group, I got the error message
Trying to prune the registry because it's full
apply: registry max-size limit reached
Debugger entered: ((cl-assertion-failed (not (registry-full db)) "registry max-size limit reached"))
cl--assertion-failed((not (registry-full db)) "registry max-size limit reached" nil nil)
registry-insert([eieio-class-tag--registry-db "~/.gnus.d/.gnus.registry.eieio" 0.2 10000 0.1...])
gnus-registry-register-message-ids()
run-hooks(gnus-summary-prepare-hook)
apply(run-hooks gnus-summary-prepare-hook)
gnus-run-hooks(gnus-summary-prepare-hook)
gnus-summary-prepare()
gnus-summary-read-group-1("nnimap+Fastmail:INBOX.mailinglists.clojure" nil t nil nil nil)
gnus-summary-read-group("nnimap+Fastmail:INBOX.mailinglists.clojure" nil t nil nil nil nil)
gnus-group-read-group(nil t)
gnus-group-select-group(nil)
gnus-topic-select-group(nil)
funcall-interactively(gnus-topic-select-group nil)
call-interactively(gnus-topic-select-group nil nil)
command-execute(gnus-topic-select-group)
My registry-related settings are just
(setq gnus-registry-ignored-groups '(("^nntp" t) ("^nndraft" t)
("^nnir" t)
("training" t) ("Junk" t)
("Trash" t) ("Spam" t))
;; Don't track anything except for the message ids.
gnus-registry-track-extra nil
gnus-registry-max-entries 10000)
(gnus-registry-initialize)
I had thought that with a maximum number of entries, as soon as I hit
it, the registry would prune the oldest entries to regain some space.
But according to the messages, it seems that pruning didn't succeed.

This is related to a change I made recently. What used to happen was --
assuming max entries of 10000 and a prune factor of 0.1 -- every time
the registry exceeded 9000 (not 10000), it would prune. That wasn't what
was supposed to happen: it was supposed to hit 10000 and then prune back
to 9000.

A few days ago I changed it so that it really did reach its max size
before pruning. Obviously something isn't quite aligned right. I'm
assuming some sort of off-by-one error: we're using #'< where we should
be using #'<=, or something like that.

Hang on a second, and I'll test this again...

Eric

Eric Abrahamsen

2015-03-30 07:15:11 UTC

Permalink

Post by Eric Abrahamsen

This is related to a change I made recently. What used to happen was --
assuming max entries of 10000 and a prune factor of 0.1 -- every time
the registry exceeded 9000 (not 10000), it would prune. That wasn't what
was supposed to happen: it was supposed to hit 10000 and then prune back
to 9000.
A few days ago I changed it so that it really did reach its max size
before pruning. Obviously something isn't quite aligned right. I'm
assuming some sort of off-by-one error: we're using #'< where we should
be using #'<=, or something like that.

Embarrassingly, that is indeed what was happening. Would you mind giving
this patch a whirl? If it works correctly, I'll push it (plus a
ChangeLog notice).

Sorry about that,

Eric

Tassilo Horn

2015-03-30 09:30:21 UTC

Permalink

Eric Abrahamsen <***@ericabrahamsen.net> writes:

Hi Eric,

Post by Eric Abrahamsen

Post by Eric Abrahamsen
A few days ago I changed it so that it really did reach its max size
before pruning. Obviously something isn't quite aligned right. I'm
assuming some sort of off-by-one error: we're using #'< where we
should be using #'<=, or something like that.

Embarrassingly, that is indeed what was happening. Would you mind
giving this patch a whirl? If it works correctly, I'll push it (plus a
ChangeLog notice).

I've switched back to 10000 max entries, and now

(registry-full gnus-registry-db)

returns t. I can enter groups and read messages without an error but I
don't get any messages about pruning. But after reading a bunch of new
messages now, (registry-full gnus-registry-db) gives nil and
(registry-size gnus-registry-db) says 9007, so apparently it has pruned
anyhow.

Bye,
Tassilo

Eric Abrahamsen

2015-03-30 09:45:47 UTC

Permalink

Post by Tassilo Horn
Hi Eric,

Post by Eric Abrahamsen

Embarrassingly, that is indeed what was happening. Would you mind
giving this patch a whirl? If it works correctly, I'll push it (plus a
ChangeLog notice).

I've switched back to 10000 max entries, and now
(registry-full gnus-registry-db)
returns t. I can enter groups and read messages without an error but I
don't get any messages about pruning. But after reading a bunch of new
messages now, (registry-full gnus-registry-db) gives nil and
(registry-size gnus-registry-db) says 9007, so apparently it has pruned
anyhow.

Okay, thanks. My guess is that the "Trying to prune the registry because
it's full" message is getting buried in other messages -- I don't see
how it could not be called. If you're really idle at the moment, could
you look in *Messages* and confirm that? But if you cut back to 10000
max entries and now have 9007, then things are almost certainly working
correctly.

Later on I'll make a new defvar (something like
`gnus-registry-pruning-functions') where third-party packages can
register pruning functions. That way, if a package registers a
"precious" key that cause entries to hang around, it can also register a
function for cleaning up after itself.

Thanks,
Eric

Tassilo Horn

2015-03-30 11:02:40 UTC

Permalink

Post by Eric Abrahamsen

Post by Tassilo Horn

Post by Eric Abrahamsen
Embarrassingly, that is indeed what was happening. Would you mind
giving this patch a whirl? If it works correctly, I'll push it (plus
a ChangeLog notice).

I've switched back to 10000 max entries, and now
(registry-full gnus-registry-db)
returns t. I can enter groups and read messages without an error but I
don't get any messages about pruning. But after reading a bunch of new
messages now, (registry-full gnus-registry-db) gives nil and
(registry-size gnus-registry-db) says 9007, so apparently it has pruned
anyhow.

I've been there and searched for "prune" without any matches.

Post by Eric Abrahamsen
But if you cut back to 10000 max entries and now have 9007, then
things are almost certainly working correctly.

Yes, seems so.

Post by Eric Abrahamsen
Later on I'll make a new defvar (something like
`gnus-registry-pruning-functions') where third-party packages can
register pruning functions. That way, if a package registers a
"precious" key that cause entries to hang around, it can also register
a function for cleaning up after itself.

Sounds good.

Bye,
Tassilo

Ted Zlatanov

2015-04-17 19:48:16 UTC

Permalink

On Mon, 30 Mar 2015 17:45:47 +0800 Eric Abrahamsen <***@ericabrahamsen.net> wrote:

EA> Later on I'll make a new defvar (something like
EA> `gnus-registry-pruning-functions') where third-party packages can
EA> register pruning functions. That way, if a package registers a
EA> "precious" key that cause entries to hang around, it can also register a
EA> function for cleaning up after itself.

Eric, thank you so much for dealing with the pruning stuff and working
on the Gnus registry.

I am not sure if such a facility will be necessary. What packages would
use it? Do we have more than one such package?

Ted

Eric Abrahamsen

2015-04-18 03:22:38 UTC

Permalink

Post by Ted Zlatanov
EA> Later on I'll make a new defvar (something like
EA> `gnus-registry-pruning-functions') where third-party packages can
EA> register pruning functions. That way, if a package registers a
EA> "precious" key that cause entries to hang around, it can also register a
EA> function for cleaning up after itself.
Eric, thank you so much for dealing with the pruning stuff and working
on the Gnus registry.
I am not sure if such a facility will be necessary. What packages would
use it? Do we have more than one such package?

Probably not :) It's mostly theoretical at this point. Though it is
entirely possible that precious keys could fill up the registry and
prevent opening of Summary buffers. This could be saved for a (very)
rainy day, or left until someone actually complains about it.

E