The internet elders will tell you not to run your own mail server. They’re not entirely wrong, but they’re not right either. Their advice comes from an era of manual compilation, cryptic sendmail configs, and open relays spewing spam across the web. It’s 2026. We have containers, better tooling, and a healthy distrust for Big Tech’s data-hungry “free” services.
Running your own mail isn’t about saving a few bucks a month. It’s about control, privacy, and learning a fundamental protocol of the internet. The complexity has shifted. The software isn’t the hard part anymore; deliverability is. This guide will show you how to tackle both.
The Unbreakable Rules of Self-Hosted Mail
Forget everything else for a moment. If you ignore these three points, your project is dead on arrival.
- You need your own domain name. This is non-negotiable.
you@yourdomain.comis the entire point. - You cannot use your home’s residential IP address. However “static” your ISP says it is. It’s on a block list somewhere, it has no reverse DNS (rDNS) entry you can control, and major providers will reject your mail on sight.
- You need a server with a clean, static IP. This means a cheap VPS from a provider like Hetzner, Vultr, or DigitalOcean. A $5/month box is more than enough. This will be your mail server’s public-facing address.
The simplest path is to run your entire mail stack on this VPS. You can also set up a VPN like WireGuard between your homelab and the VPS, using the VPS as a mail gateway, but that adds another point of failure. Start with the VPS.
The Modern Mail Stack, Containerized
We aren’t going to apt install a dozen packages and pray they work together. We’re using Docker. While you could assemble individual Postfix, Dovecot, and Rspamd containers, a few projects have already done the hard work of integrating them.
The docker-mailserver project is a solid choice. It bundles all the necessary components into a single, well-documented container image. It’s not a black box; all the configuration is exposed through environment variables and config files mounted via Docker volumes, giving you the control you need without the initial setup nightmare.
Here’s a basic docker-compose.yml to get you started. Save this on your VPS.
services:
mailserver:
image: ghcr.io/docker-mailserver/docker-mailserver:latest
container_name: mailserver
hostname: mail.yourdomain.com # This MUST match your MX record's target
env_file: mailserver.env
ports:
- "25:25" # SMTP
- "143:143" # IMAP (STARTTLS)
- "465:465" # SMTPS
- "587:587" # SMTP Submission (for clients)
- "993:993" # IMAPS
volumes:
- ./docker-data/dms/mail-data/:/var/mail/
- ./docker-data/dms/mail-state/:/var/mail-state/
- ./docker-data/dms/mail-logs/:/var/log/mail/
- ./docker-data/dms/config/:/tmp/docker-mailserver/
- /etc/localtime:/etc/localtime:ro
- /etc/letsencrypt:/etc/letsencrypt:ro # certificates for SSL_TYPE=letsencrypt
restart: always
stop_grace_period: 1m
cap_add:
- NET_ADMIN # Needed for fail2ban to work correctly
You’ll also need a mailserver.env file to hold your configuration:
# mailserver.env (a subset; start from the project's full mailserver.env)
# Warning: Do not use quotes around values!
# Use certificates from /etc/letsencrypt (obtained with certbot or similar
# for the container's hostname, e.g. mail.yourdomain.com)
SSL_TYPE=letsencrypt
POSTMASTER_ADDRESS=postmaster@yourdomain.com
# Rspamd for spam filtering, DKIM signing and SPF/DMARC checks.
# The project recommends disabling the older components it replaces:
ENABLE_RSPAMD=1
ENABLE_OPENDKIM=0
ENABLE_OPENDMARC=0
ENABLE_POLICYD_SPF=0
ENABLE_AMAVIS=0
# Virus scanning (needs extra RAM) and brute-force protection
ENABLE_CLAMAV=1
ENABLE_FAIL2BAN=1
Start the container and create your first account right away (docker-mailserver expects at least one account shortly after first start):
docker compose up -d
docker exec -ti mailserver setup email add contact@yourdomain.com
# you'll be prompted for the password, so it doesn't end up in your shell history
Now, bring the stack online. It will automatically fetch a Let’s Encrypt certificate for mail.yourdomain.com.
The DNS Grind: Proving You’re Not a Spammer
This is where most people fail. Your software can be perfect, but without correct DNS records, your emails go straight to spam. Log into your domain registrar’s DNS panel and get ready.
- A Record: Point
mail.yourdomain.comto your VPS’s static IP address.mail IN A 198.51.100.10
- MX Record: Tell the world that
mail.yourdomain.comhandles mail foryourdomain.com. The10is the priority.yourdomain.com. IN MX 10 mail.yourdomain.com.
- PTR Record (rDNS): This is the reverse of an A record. You set this in your VPS provider’s control panel, not your DNS registrar. You want your IP address to resolve back to your mail server’s hostname.
198.51.100.10->mail.yourdomain.com
- SPF Record: A TXT record that lists which servers are allowed to send mail for your domain.
yourdomain.com. IN TXT "v=spf1 mx -all"(This says “only the server listed in my MX record can send mail.”)
- DKIM Record: A cryptographic signature to verify the sender.
docker-mailservergenerates this for you.- Run:
docker exec -ti mailserver setup config dkim, then publish the generated TXT record shown in the output (or indocker-data/dms/config/). - Copy the output, which will be a TXT record for
mail._domainkey.yourdomain.com, and add it to your DNS.
- Run:
- DMARC Record: A policy that tells receiving servers what to do if SPF or DKIM fails. Start with a reporting-only policy to avoid accidentally blocking your own mail.
_dmarc.yourdomain.com. IN TXT "v=DMARC1; p=none; rua=mailto:postmaster@yourdomain.com"
After setting these, wait. DNS propagation can take a while. Use a tool like MXToolbox to verify everything is correct before you proceed.
Living With It: The Realities of Self-Hosting
Your server is running and your DNS is set. Now the real work begins.
The IP Warm-Up: Your new IP has no reputation. Don’t immediately try to send 1,000 emails. Start by sending a few emails to accounts you control on Gmail, ProtonMail, etc. Open them, click links, reply. This builds a positive sending history.
Microsoft is the Final Boss: Getting email delivered to outlook.com and hotmail.com is notoriously difficult. Their filters are aggressive and opaque. If your mail is rejected, check the bounce message for a link to their delisting form. Be polite, persistent, and patient. It can take weeks.
Monitor Your Logs: If mail isn’t being delivered, the logs are your only source of truth. Use docker compose logs -f mailserver to see the live transaction logs. Rejection messages are often cryptic but contain the clues you need.
Backups, Backups, Backups: Your docker-data directory contains all your mail and configuration. Back it up regularly. A simple tar command in a cron job, pushing the archive to another location (like an S3 bucket or your NAS), is a good start.
Self-hosting your email is a commitment. It requires attention. But the payoff is a completely private communication channel that you control from end to end. You’ll understand the internet’s infrastructure on a deeper level and be free from third-party data mining. It’s not for everyone, but if you’re a homelab enthusiast, it’s one of the most rewarding projects you can undertake.
What’s Next
Once your mail server is stable, you can integrate it further into your homelab.
- Set Up a Secure Homelab Gateway with WireGuard and a VPS
- Monitor Your Services with Prometheus and Grafana
- Automate Server Configuration with Ansible Basics
- Build a Blazing Fast Blog with Astro
Get notified when new articles and designs land:
No spam. Unsubscribe any time.
[discussion]
Comments are powered by Giscus — backed by GitHub Discussions. Sign in with GitHub to join the conversation.