Mailman3 on Debian (with exim4)
Overview
We've been using Mailman to host our mailing lists for decades. Since Mailman2 has become obsolete a while ago, it's time to move to Mailman3.
Setting up a fresh Mailman3 instance
Installing Mailman3 on Debian is pretty easy, just apt-get install mailman3-full.
Of course this misses the 2nd most important thing: the database backend!
Ideally one should probably start out with installing postgresql before trying to install mailman3-full.
I did it the other way round.
So after I corrected the error, I ran
1dpkg-reconfigure mailman3
2dpkg-reconfigure mailman3-web
and answered all the questions to the best of my knowledge.
getting rid of the errors
After the initial setup, I started receiving emails from Mailman3. Great (at least the mail is working)! Unfortunately, the mails contained error messages, and they were generated every minute!
AutoField
There was an error at the end of reconfiguring mailman3-web, which I ignored, but it came back to me.
In an email.
Every single minute.
1System check identified some issues:
2
3WARNINGS:
4django_mailman3.MailDomain: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
5 HINT: Configure the DEFAULT_AUTO_FIELD setting or the DjangoMailman3Config.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
So where and how to set this field?
After long searching, I eventually found that i can just add it to the end of /etc/mailman3/mailman-web.py:
1DEFAULT_AUTO_FIELD='django.db.models.AutoField'
"Retry and Timeout are misconfigured"
Unfortunately this was only part of it, as now I got another error mail:
1/usr/lib/python3/dist-packages/django_q/conf.py:139: UserWarning: Retry and timeout are misconfigured. Set retry larger than timeout,
2 failure to do so will cause the tasks to be retriggered before completion.
3 See https://django-q.readthedocs.io/en/latest/configure.html#retry for details.
4 warn(
After a bit of digging in the source code, it turned out that /etc/mailman3/mailman-web.py
is missing some crucial parts: namely a Q_CLUSTER configuration.
At first I was stymied, because at the top of the config file it says:
1# This file is imported by the Mailman Suite. It is used to override
2# the default settings from /usr/share/mailman3-web/settings.py.
and I understood this to mean, that one can override specific values otherwise defined in /usr/share/mailman3-web/settings.py.
Which turned out to be not true.
Instead, this file is a substitute of the "default" file, so any settings in the /usr/share file are for reference but do not take any effect.
To cut a long story short, I needed to manually add retry and timeout values to the Q_CLUSTER dict (but read the next section, as this is not the end of it):
1Q_CLUSTER = {
2 'retry': 360,
3 'timeout': 300,
4}
(according to /usr/lib/python3/dist-packages/django_q/conf.py, retry defaults to 60, and timeout is unset by default,
the latter is the reason for the error)
deleting archives
After that, the errors disappeared. So I played around a little bit, and imported and old Mailman2 mailinglist including archives, which went fine. I then went on to delete the archives of my test list via the webinterface, and was again greeted with an error on the webiterface, and a longish error mail in my INBOX.
The mail starts with:
1Internal Server Error: /mailman3/hyperkitty/list/test-list@lists.example.org/delete/
2
3ConnectionError at /hyperkitty/list/test-list@lists.example.org/delete/
4Error 111 connecting to localhost:6379. Connection refused.
Port 6379 is used by Redis, but I didn't have a redis server installed. Did I miss something?
In The end I found, that Mailman3 supports multiple task queue backends,
and the default being obviously Redis.
Luckily, the documentation comes with an alternative suggestion (namely orm), and so the full Q_CLUSTER setup in /etc/mailman3/mailman-web.py
looks like this:
1Q_CLUSTER = {
2 'retry': 360,
3 'timeout': 300,
4 'save_limit': 100,
5 'orm': 'default',
6}
minor cleanups
In /etc/mailman3/mailman.cfg is also set:
1site_owner: admin@example.org
In /etc/mailman3/mailman-web.py, I changed the ADMINS to have my email address (admin@example.org) instead of the default.
(This was so I could log into the web interface as administrator; after setting the email address here,
I tried to login with that email and used the "Forgotten password" functionality to set my initial password.
Probably there's a cmdline way to initialize the admin).
I also changed the EMAILNAME to lists.example.org.
Finally, I uncommented the 'django_mailman3.lib.auth.fedora' entry from the INSTALLED_APPS, as I don't see why they should be allowed to login.
(and don't forget about the Q_CLUSTER and DEFAULT_AUTO_FIELD mentioned above).
configuring exim4
(NOTE: These configs are basically taken directly from the Mailman documentation)
The default mailserver in Debian is exim4, which is what I'm going to use.
The Debian package for Mailman3 enabled postfix support by default, which made mailman fail to start.
Luckily the that is an easy to spot error, so I made sure that /etc/mailman3/mailman.cfg contained the following
values in the [mta] section:
1[mta]
2incoming: mailman.mta.exim4.LMTP
3outgoing: mailman.mta.deliver.deliver
4smtp_host: localhost
5smtp_port: 25
6smtp_user:
7smtp_pass:
8lmtp_host: 127.0.0.1
9lmtp_port: 8024
10configuration: python:mailman.config.exim4
Every installation of differs slightly. Hence it is a good idea, to put storage paths and the like into a central place.
For our exim4 configuration, we put all these variables into a single
25_mm3_macros file at the beginning of the configuration.
For a Debian installation of Mailman3, most things will be pretty much the same,
with the exception of the mm_domains list, which lists all domains for your mailinglists:
1# /etc/exim4/conf.d/main/25_mm3_macros
2# The colon-separated list of domains served by Mailman.
3domainlist mm_domains=lists.example.com:lists.example.org
4
5# same as in /etc/mailman3/mailman.cfg:
6MM3_LMTP_PORT=8024
7
8# MM3_HOME must be set to mailman's var directory, wherever it is
9# according to your installation.
10MM3_HOME=/var/lib/mailman3
11MM3_UID=list
12MM3_GID=list
13
14################################################################
15# The configuration below is boilerplate:
16# you should not need to change it.
17
18# The path to the list receipt (used as the required file when
19# matching list addresses)
20MM3_LISTCHK=MM3_HOME/lists/${local_part}.${domain}
So now we need to tell exim4, that for each mailinglist address,
it should use the mailman3_transport.
The check whether an email should go to a mailman (rather than - say - an inbox),
is based on the domain (which must be in the mm_domains list) and the existence
of some file on the harddisk, that is named list the mailinglist address,
as defined by the MM3_LISTCHK macro (e.g. here, mailman will create a file
/var/lib/mailman3/lists/mylist.lists.example.org for the <mylist@lists.example.org>
mailinglist, which makes it easy to check by exim:
1# /etc/exim4/conf.d/router/455_mm3_router
2.ifdef MM3_HOME
3
4mailman3_router:
5 driver = accept
6 domains = +mm_domains
7 require_files = MM3_LISTCHK
8 local_part_suffix_optional
9 local_part_suffix = \
10 -bounces : -bounces+* : \
11 -confirm : -confirm+* : \
12 -join : -leave : \
13 -owner : -request : \
14 -subscribe : -unsubscribe
15 transport = mailman3_transport
16
17.endif
Finally, the mailman3_transport delivers the mails selected by the mailman3_router
via lmtp to Mailman:
1# /etc/exim4/conf.d/transport/55_mm3_transport
2.ifdef MM3_HOME
3
4mailman3_transport:
5 driver = smtp
6 protocol = lmtp
7 allow_localhost
8 hosts = localhost
9 port = MM3_LMTP_PORT
10 rcpt_include_affixes = true
11
12.endif
mail archives
One of the nice things about Mailman3 is, that we can add permalinks to the archived mails.
For this to work, the archiver (hyperkitty) and the webfrontend must share a secret API key.
For the archiver, this is set in /etc/mailman3/mailman-hyperkitty.cfg:
1[general]
2api_key: SomeVerySecretKey
And for the frontend, the same key must be set in /etc/mailman3/mailman-web.py:
1MAILMAN_ARCHIVER_KEY = 'SomeVerySecretKey`
It seems that the Debian setup already created a random key for the front-end.
However, the archiver was set to the default 'SecretArchiverAPIKey'.
So I just had to copy the API key from mailman-web.py to mailman-hyperkitty.cfg.
With this, mails sent through Mailman will automatically gain a Archived-At header (RFC 5064).
One can also add a custom footer (via the webinterface) that shows the permalink:
1_______________________________________________
2$display_name mailing list -- $listname
3To unsubscribe send an email to ${short_listname}-leave@${domain}
4This mail is archived under ${hyperkitty_url}
Importing old mailinglists from mailman2
In theory everything is described in the Mailman3 docs.
Where are those commands?
In practice, the docs speak of commands that I can (or should) not execute.
1mailman import21 foo-list@lists.example.org /path/to/mailman2/foo-list/config.pck
As the documentation notes, the mailman command should not be run by root, but instead by the mailman user.
Debian provides a /usr/bin/mailman-wrapper script, that can be used instead (as root) and is executed as the list user.
So instead, the correct command should be:
1mailman-wrapper import21 foo-list@lists.example.org /path/to/mailman2/foo-list/config.pck
The 2nd command the docs speak about, gave me more trouble:
1python manage.py hyperkitty_import -l foo-list@lists.example.org $var_prefix/archives/private/foo-list.mbox/foo-list.mbox
there's no manage.py anywhere to be found (and Mailman3 is spread over multiple binary packages, so it is a bit hard to search).
In the very end, I found that it lives in /usr/share/mailman3-web/manage.py (and must not be confused with /usr/share/doc/python3-django-hyperkitty/examples/manage.pywww-data user!),
I also discovered, that in Debian there is /usr/bin/mailman-web which does exactly that.
So we can now import old list archives via:
1mailman-web hyperkitty_import -l foo-list@lists.example.org $var_prefix/archives/private/foo-list.mbox/foo-list.mbox
2mailman-web update_index_one_list foo-list@lists.example.org
Checking for mbox consistency issues (before importing)
Sometimes the mbox file for importing a list archive can be a bit broken.
Mailman provides some utilities to check this beforehand, but unfortunately, these have been dropped from the Debian packages (for whatever reasons).
You can download them with
1wget https://salsa.debian.org/mailman-team/hyperkitty/-/raw/master/hyperkitty/contrib/cleanarch3
2wget https://salsa.debian.org/mailman-team/hyperkitty/-/raw/master/hyperkitty/contrib/check_hk_import
3chmod +x cleanarch3 check_hk_import
And then run them like so:
1./cleanarch3 -n $var_prefix/archives/private/foo-list.mbox/foo-list.mbox
2./check_hk_import $var_prefix/archives/private/foo-list.mbox/foo-list.mbox
fix mbox encoding
Sometimes, single messages in my mbox files have a wrong encoding.
1#!/usr/bin/env python3
2import sys
3
4codecs = [ "iso8859_15",
5 "iso8859_2", "iso8859_3", "iso8859_4", "iso8859_5", "iso8859_6", "iso8859_7", "iso8859_8",
6 "iso8859_9", "iso8859_10", "iso8859_11", "iso8859_13", "iso8859_14", "iso8859_16",
7 "big5", "big5hkscs", "cp037", "cp273", "cp424", "cp437", "cp500", "cp720", "cp737", "cp775",
8 "cp850", "cp852", "cp855", "cp856", "cp857", "cp858", "cp860", "cp861", "cp862", "cp863",
9 "cp864", "cp865", "cp866", "cp869", "cp874", "cp875", "cp932", "cp949", "cp950", "cp1006",
10 "cp1026", "cp1125", "cp1140", "cp1250", "cp1251", "cp1252", "cp1253", "cp1254", "cp1255",
11 "cp1256", "cp1257", "cp1258", "euc_jp", "euc_jis_2004", "euc_jisx0213", "euc_kr", "gb2312",
12 "gbk", "gb18030", "hz", "iso2022_jp", "iso2022_jp_1", "iso2022_jp_2", "iso2022_jp_2004",
13 "iso2022_jp_3", "iso2022_jp_ext", "iso2022_kr", "latin_1", "johab", "koi8_r", "koi8_t",
14 "koi8_u", "kz1048", "mac_cyrillic", "mac_greek", "mac_iceland", "mac_latin2", "mac_roman",
15 "mac_turkish", "ptcp154", "shift_jis", "shift_jis_2004", "shift_jisx0213", "utf_32",
16 "utf_32_be", "utf_32_le", "utf_16", "utf_16_be", "utf_16_le", "utf_7", "utf_8_sig",
17]
18
19def fixcoding(filename):
20 with open(filename, "rb") as f:
21 data = f.read()
22 d2 = None
23 try:
24 # already UTF-8
25 data.decode("utf-8")
26 sys.stdout.buffer.write(data)
27 return
28 except:
29 pass
30 for codec in codecs:
31 try:
32 # try to convert from a different encoding
33 d2 = data.decode(codec).encode("utf-8")
34 sys.stdout.buffer.write(d2)
35 return
36 except:
37 pass
38 # give up
39 sys.stdout.buffer.write(data)
40
41for f in sys.argv[1:]:
42 fixcoding(f)
We want to run this on each message in an mbox separately:
1#!/bin/sh
2mbox="$1"
3splitdir=$(mktemp -d)
4
5git mailsplit -o"${splitdir}" -d7 --keep-cr "${mbox}" 1>&2
6find "${splitdir}/" -maxdepth 1 -type f -print0 | sort -zV | xargs -0 iso2utf8.py | cleanarch3 /dev/stdin
7rm -rf "${splitdir}"
Non-ASCII characters
Some of our users have outlandish names with non-ASCII characters.
import21 chokes!
1$ mailman-wrapper import21 foo-dev@lists.example.org /var/lib/mailman/lists/foo-dev/config.pck
2Importing members [####--------------------------------] 11% 00:00:15
3/usr/lib/python3/dist-packages/mailman/database/base.py:60: SAWarning: Session's state has been changed on a non-active transaction - this state will be discarded.
4 self.store.rollback()
5Traceback (most recent call last):
6 File "/usr/bin/mailman", line 33, in <module>
7 sys.exit(load_entry_point('mailman==3.3.8', 'console_scripts', 'mailman')())
8 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
9[...]
10 File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
11 self.dialect.do_execute(
12 File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 736, in do_execute
13 cursor.execute(statement, parameters)
14UnicodeEncodeError: 'ascii' codec can't encode character '\xe3' in position 2: ordinal not in range(128)
...
IIRC, I ended up manually changing the names of the few affected users.
Re-linking the archives
mailman2 archived messages on disk, using a date-based naming scheme like pipermail/foo-list/2024-02/123456.html.
mailman3 uses a different approach, where each mail gets a unique ID
(calculated by hashing the message-ID, and probably adding some salt to avoid duplicate message-IDs from adversarial posters),
resulting in something like hyperkitty/list/foo-dev@lists.example.org/message/WSNQP5OPATMAWXTBGGL3NIPAO4YWATU2.
I would like to not break the old archive links, and have them redirect to the new shiny (interactive, and whatnot) hyperkitty archives.
The handle for this are the Message-IDs found in each email,
as they are recorded both the mailman3 database and in the mailman2 HTML archive files.
To get the mm3 mapping from the Message-IDs to the hash, we can query the DB:
1echo 'SELECT message_id, message_id_hash, subject, date, sender_name FROM hyperkitty_email;' | psql --csv mailman3web.db > mm3ids.csv
In the mm2 archive, the message ID is hidden in the "Reply" link. We can simply parse:
1from bs4 import BeautifulSoup
2import urllib.parse
3
4def getMessageID(filename):
5 with open(filename) as f:
6 soup = BeautifulSoup(f.read(), "html.parser")
7 ids = []
8 for link in soup.head.findChildren("link", rel="made"):
9 url = link["href"]
10 query = urllib.parse.parse_qs(urllib.parse.urlsplit(url).query)
11 ids += [_.lstrip("<").rstrip(">") for _ in query.get("In-Reply-To", [])]
12 return ids
...
The above was somehow used to create HTML-pages that would re-direct the old mm2 archives to the new mm3 ones, by reading the message-ID from each mm2-archived page and looking up the hash for the message-ID in the CSV file.
1<!DOCTYPE HTML>
2<html lang="en-US">
3 <head>
4 <meta charset="UTF-8">
5 <meta http-equiv="refresh" content="0; url=https://lists.example.org/hyperkitty/list/foo-dev@lists.iem.at/message/@HASH@">
6 <script type="text/javascript">window.location.href = "https://lists.example.org/hyperkitty/list/foo-dev@lists.example.org/message/@HASH@"</script>
7 <title>List Archive Redirection</title>
8 </head>
9 <body>
10 If you are not redirected automatically, follow this <a href='https://lists.example.org/hyperkitty/list/foo-dev@lists.iem.at/message/@HASH@'>link to the new archive</a>.
11 </body>
12</html>
🤦
OMG. This is where my notes end. This documented was originally started in summer 2024 when I did the migration to Mailman3. I successfully finished the migration, but not the documentation :-(