mail-header-separator starting with whitespace fails

[Russ Allbery]

Hello all,

I just upgraded to Emacs 24 from Emacs 23, with the corresponding change
in Gnus versions. For the most part this went smoothly (thank you!). But
there was one rather annoying hiccup that caused some problems, and I
thought I'd mention it somewhere so that people could consider whether
it's worth fixing.

I've never liked the default mail-header-separator in Emacs. I think it
blends in with the header and the start of the body text. So for years
I've used:

(setq mail-header-separator (concat (make-string 39 ? ) "."))

to create a separator that's much less obtrusive.

Unfortunately, when upgrading from Emacs 23 to Emacs 24, something in Gnus
or in the underneath modes changed so that this mail-header-separator no
longer works. It's appended to the Content-Type header (causing message
corruption if it has a charset setting), and then the first line of the
body is lifted into the headers and disappears.

Removing this setting fixes the problem, but brings back the default
message separator.

I did a little bit of poking around, and it looks like something inside
the Gnus message code isn't actually using mail-header-separator but
instead is looking for the first line in the buffer that doesn't look like
a header. Since my message separator looked like a continuation line, it
skipped right past it and kept going. This seems surprising to me. I
think whatever is in mail-header-separator should always work, even if it
looks like a header.

[Katsumi Yamaoka]

Post by Russ Allbery
(setq mail-header-separator (concat (make-string 39 ? ) "."))

I tried this, confirmed it causes a mail corruption, and found
at least one cause. Could you try replacing the function definition
of `rfc822-goto-eoh' with this? (I.e., eval this form?)

--8<---------------cut here---------------start------------->8---
(defun rfc822-goto-eoh ()
"If the buffer starts with a mail header, move point to the header's end.
Otherwise, moves to `point-min'.
The end of the header is the start of the next line, if there is one,
else the end of the last line. This function obeys RFC822."
(goto-char (point-min))
(when (re-search-forward
(concat "^" (regexp-quote mail-header-separator) "\n\\|"
"^\\([:\n]\\|[^: \t\n]+[ \t\n]\\)")
nil 'move)
(goto-char (match-beginning 0))))
--8<---------------cut here---------------end--------------->8---

This function is used by `smtpmail-send-it' by way of
`mail-sendmail-undelimit-header' to delete the separator, but
the original one doesn't respect `mail-header-separator'.

But I'm not quite sure that it is a cause of your problem, since
those functions seem not to have been modified for years. Didn't
you use a separator that does not begin with a SPC, like this?

. .

[Russ Allbery]

Post by Katsumi Yamaoka

Post by Russ Allbery
(setq mail-header-separator (concat (make-string 39 ? ) "."))

I tried this, confirmed it causes a mail corruption, and found
at least one cause. Could you try replacing the function definition
of `rfc822-goto-eoh' with this? (I.e., eval this form?)
(defun rfc822-goto-eoh ()
"If the buffer starts with a mail header, move point to the header's end.
Otherwise, moves to `point-min'.
The end of the header is the start of the next line, if there is one,
else the end of the last line. This function obeys RFC822."
(goto-char (point-min))
(when (re-search-forward
(concat "^" (regexp-quote mail-header-separator) "\n\\|"
"^\\([:\n]\\|[^: \t\n]+[ \t\n]\\)")
nil 'move)
(goto-char (match-beginning 0))))
This function is used by `smtpmail-send-it' by way of
`mail-sendmail-undelimit-header' to delete the separator, but
the original one doesn't respect `mail-header-separator'.

Sorry about the delay in verifying this. That change does indeed fix my
problem. Thank you!

Post by Katsumi Yamaoka
But I'm not quite sure that it is a cause of your problem, since
those functions seem not to have been modified for years. Didn't
you use a separator that does not begin with a SPC, like this?
. .

No, the mail-header-separator setting I was using had been unchanged since
I started using Gnus back in 1994.

I was previously using the Gnus included in Emacs 23.4. Something must
have changed from that version to Emacs 24.3 that caused this corruption,
although I'm not sure what.

[Lars Ingebrigtsen]

Post by Katsumi Yamaoka
I tried this, confirmed it causes a mail corruption, and found
at least one cause. Could you try replacing the function definition
of `rfc822-goto-eoh' with this? (I.e., eval this form?)
(defun rfc822-goto-eoh ()
"If the buffer starts with a mail header, move point to the header's end.
Otherwise, moves to `point-min'.
The end of the header is the start of the next line, if there is one,
else the end of the last line. This function obeys RFC822."
(goto-char (point-min))
(when (re-search-forward
(concat "^" (regexp-quote mail-header-separator) "\n\\|"
"^\\([:\n]\\|[^: \t\n]+[ \t\n]\\)")
nil 'move)
(goto-char (match-beginning 0))))
This function is used by `smtpmail-send-it' by way of
`mail-sendmail-undelimit-header' to delete the separator, but
the original one doesn't respect `mail-header-separator'.

The mail header separator should have been stripped before the message
was passed to smtpmail, though, so I think something else must be wrong
somewhere...

[Katsumi Yamaoka]

Post by Lars Ingebrigtsen

Post by Katsumi Yamaoka
(defun rfc822-goto-eoh ()

[...]

Post by Lars Ingebrigtsen

Post by Katsumi Yamaoka
This function is used by `smtpmail-send-it' by way of
`mail-sendmail-undelimit-header' to delete the separator, but
the original one doesn't respect `mail-header-separator'.

The mail header separator should have been stripped before the message
was passed to smtpmail, though, so I think something else must be wrong
somewhere...

Gnus seems to have been not deleting the mail-header-separator
before passing a message to message-send-mail-function for over
ten years. I've verified it using message-mail of Ma Gnus v0.1,
No Gnus v0.1, and Oort Gnus v0.01 (2001-04-14). ;-)

[Lars Ingebrigtsen]

Post by Katsumi Yamaoka
Gnus seems to have been not deleting the mail-header-separator
before passing a message to message-send-mail-function for over
ten years. I've verified it using message-mail of Ma Gnus v0.1,
No Gnus v0.1, and Oort Gnus v0.01 (2001-04-14). ;-)

Heh.

I've now made message.el remove the separator when using smtpmail, too.
That should probably fix the problem the user had.

Should perhaps be backported to emacs-24 if this doesn't lead to any
problems.

[Katsumi Yamaoka]

Post by Lars Ingebrigtsen
I've now made message.el remove the separator when using smtpmail, too.
That should probably fix the problem the user had.
Should perhaps be backported to emacs-24 if this doesn't lead to any
problems.

Thanks. That shouldn't cause any problem, I believe. So, I've
merged it to both the trunk and the emacs-24 branch.