Go backSelf-Hosted SMTP Server on Ubuntu VPS with Mailcow, Nginx and Docker
May 03, 202612 minutes read
Introduction
Self-hosting your own mail server means taking back control over your communications, reducing your dependency on third-party providers, and natively integrating email sending into your own services (monitoring, notifications, applications). In this guide, we'll set up a complete and robust stack on an Ubuntu VPS with:
-
Mailcow
— all-in-one mail stack (Postfix, Dovecot, Rspamd, SOGo webmail)
-
Docker
— all services are containerized
-
Native Nginx
— reverse proxy with TLS via Certbot
-
A shared Docker network
— so other services on the VPS can send emails
Prerequisites
- Ubuntu 22.04+ VPS
- A domain name (we'll use `yourdomain.com` throughout this guide)
- Nginx installed natively (`apt`)
- Certbot installed
- Docker + Docker Compose plugin installed
- Port 25 unblocked by your hosting provider (open a support ticket if needed)
1. DNS Configuration
First, configure these DNS records at your registrar:
| Type | Name | Value |
|------|------|-------|
| A | `mail` | `YOUR_VPS_IP` |
| MX | `@` | `mail.yourdomain.com` (priority 10) |
| TXT | `@` | `v=spf1 mx ~all` |
| TXT | `_dmarc` | `v=DMARC1; p=quarantine; rua=mailto:admin@yourdomain.com` |
| CNAME | `autoconfig` | `mail.yourdomain.com` |
| CNAME | `autodiscover` | `mail.yourdomain.com` |
DKIM will be generated later by Mailcow — you'll add it after installation.
2. Prepare the VPS
apt update && apt upgrade -y
# Open required ports
ufw allow 22/tcp
ufw allow 80/tcp
ufw allow 443/tcp
ufw allow 25/tcp
ufw allow 465/tcp
ufw allow 587/tcp
ufw allow 993/tcp
ufw allow 4190/tcp
ufw enable
# Disable the system Postfix if installed (conflicts with Mailcow)
systemctl stop postfix
systemctl disable postfix
3. Folder Structure
mkdir -p /opt/mailserver/{mailcow,monitoring}
cd /opt/mailserver
4. TLS Certificate with Certbot
Before enabling the HTTPS vhost, generate the certificate using webroot mode.
First, create a temporary HTTP vhost:
# /etc/nginx/sites-available/mailcow-temp
server {
listen 80;
server_name mail.yourdomain.com;
root /var/www/html;
}
ln -s /etc/nginx/sites-available/mailcow-temp /etc/nginx/sites-enabled/
nginx -t && systemctl reload nginx
certbot certonly --webroot -w /var/www/html -d mail.yourdomain.com
# Verify
ls /etc/letsencrypt/live/mail.yourdomain.com/
5. Nginx Vhost — Reverse Proxy to Mailcow
# /etc/nginx/sites-available/mailcow
server {
listen 80;
listen [::]:80;
server_name mail.yourdomain.com;
# Required for Certbot renewal
location /.well-known/acme-challenge/ {
root /var/www/html;
try_files $uri =404;
}
location / {
return 301 https://$host$request_uri;
}
}
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name mail.yourdomain.com;
ssl_certificate /etc/letsencrypt/live/mail.yourdomain.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/mail.yourdomain.com/privkey.pem;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers HIGH:!aNULL:!MD5;
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
add_header X-Frame-Options SAMEORIGIN;
add_header X-Content-Type-Options nosniff;
client_max_body_size 50m;
location / {
proxy_pass http://127.0.0.1:8181;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https; # hardcoded https, not $scheme
proxy_set_header X-Forwarded-Host $host;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;
proxy_buffers 32 4k;
}
location /Microsoft-Server-ActiveSync {
proxy_pass http://127.0.0.1:8181/Microsoft-Server-ActiveSync;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_connect_timeout 75;
proxy_send_timeout 3650;
proxy_read_timeout 3650;
client_max_body_size 200m;
}
}