Self-hosted email on a VPS can be reliable, but reliability depends on more than installing Mailcow. This guide covers the deployment decisions, DNS records, authentication, mailbox migration, deliverability checks, monitoring, and maintenance routines that keep a Mailcow server usable over time.
Overview
Mailcow is a Docker-based mail platform that combines common email services behind a web administration interface. It can simplify the work of running mailboxes, aliases, spam filtering, webmail, and domain authentication, but it does not remove the operational responsibilities of email hosting.
Before deploying, confirm that your VPS provider is suitable for mail. You need a stable public IPv4 address, a provider that allows outbound SMTP traffic, control over reverse DNS, and enough storage for mailboxes, indexes, logs, and backups. A small test server may be adequate for one or two mailboxes, but storage and memory requirements grow with message retention, attachments, filtering, and the number of domains.
Use a dedicated hostname such as mail.example.com and keep the mail server's public IP stable. Do not place the SMTP service behind a generic web proxy or a tunnel designed only for HTTP applications. Mail delivery depends on direct SMTP connectivity, reverse DNS, and consistent identity.
Mailcow's installation requirements and supported operating systems can change between releases. Read the current project documentation before creating the VPS, then use a clean, supported Linux installation. Avoid installing unrelated applications on the same host until the mail stack is working and its resource requirements are understood.
Preflight checklist
- Choose a VPS with a static public IP and adequate disk space.
- Confirm outbound TCP port 25 is available, or ask the provider about its mail policy.
- Confirm that reverse DNS can be set for the server IP.
- Register the mail hostname and decide which domains the server will host.
- Install a supported operating system on a clean server.
- Apply updates, create a non-root administrator, and configure SSH key access.
- Record a backup destination that is independent of the VPS.
- Plan how you will monitor SMTP reachability, disk usage, certificates, and backups.
What to track
Mailcow deployment and TLS
Download Mailcow using the current documented installation method and review the generated configuration before starting the Docker Compose stack. Set the primary mail hostname carefully; it becomes part of the server's identity in banners, certificates, and configuration.
Publish the hostname in DNS before requesting certificates. Mailcow's web interface and mail services should use valid TLS certificates, and clients should connect using names covered by those certificates. Check certificate expiry as an operational metric rather than assuming renewal will always succeed. A failed renewal can be caused by incorrect DNS, blocked HTTP validation, or a changed hostname.
Keep the Mailcow stack isolated from other reverse proxies unless you understand the port and certificate requirements. If you already operate Traefik or Nginx Proxy Manager, document which service owns ports 80 and 443. A conflict can make the administration interface or certificate automation appear broken. For broader reverse-proxy design, see our Traefik and Nginx Proxy Manager comparison.
DNS records for a mail domain
Replace the example values below with your actual hostname, domain, IP address, and Mailcow-generated DKIM selector. Keep one authoritative record set and remove obsolete records after migration.
| Record | Example | Purpose |
|---|---|---|
| A | mail.example.com → 203.0.113.10 | Maps the mail hostname to the VPS address. |
| MX | example.com → mail.example.com | Directs inbound mail to the server. Set an explicit priority. |
| PTR | 203.0.113.10 → mail.example.com | Reverse DNS set with the VPS provider; the forward and reverse names should agree. |
| SPF TXT | example.com → v=spf1 mx -all | Declares which hosts may send for the domain. Publish one SPF record only. |
| DKIM TXT | selector._domainkey.example.com → Mailcow-provided public key | Allows recipients to verify a cryptographic signature added by the server. |
| DMARC TXT | _dmarc.example.com → v=DMARC1; p=none; rua=mailto:[email protected] | Reports authentication results and defines an initial monitoring policy. |
| Autodiscover or client records | As required by your clients and Mailcow configuration | Helps configure IMAP, SMTP submission, and webmail access consistently. |
Do not copy a DKIM key from another domain or server. Generate or retrieve the key from the Mailcow administration interface, publish the complete value as the DNS provider requires, and verify that the selector resolves publicly. SPF describes sending authorization; DKIM signs messages; DMARC evaluates alignment between those results and the visible From domain. They solve different problems and should be tested separately.
Mailboxes, spam controls, and monitoring
Create a domain, mailbox, aliases, and quotas in the Mailcow interface. Use separate aliases for public addresses such as contact or support instead of sharing one password among several people. Require strong, unique mailbox passwords and enable any available account protections appropriate to your deployment.
Review spam filtering conservatively during the first weeks. Aggressive settings can hide legitimate mail, while permissive settings increase unwanted messages. Inspect quarantine and rejected-message logs before changing thresholds. Track disk usage, queue length, delivery errors, container health, CPU and memory pressure, certificate expiry, and backup results. A monitoring stack such as Uptime Kuma, Prometheus, and Grafana can provide a useful surrounding layer, but alerting must test meaningful mail functions rather than only whether a web page responds.
Cadence and checkpoints
First 24 to 72 hours
- Resolve the mail hostname and confirm forward and reverse DNS agree.
- Send test messages to several unrelated recipient systems and inspect authentication results.
- Receive mail from an external account and verify MX routing.
- Test IMAP, SMTP submission, webmail, aliases, and password changes.
- Confirm that certificates are valid and that renewal is scheduled.
- Check logs for authentication failures, queue growth, and repeated delivery errors.
Weekly and monthly checks
Weekly, review the mail queue, disk growth, failed logins, spam quarantine, and backup completion. Investigate persistent queue entries rather than repeatedly retrying them. Confirm that monitoring alerts reach an account outside the VPS.
Monthly, apply updates according to Mailcow's release guidance and your change process. Read release notes, make a backup, verify available disk space, and update during a period when you can inspect logs afterward. Also review mailbox quotas, inactive accounts, forwarding rules, and the list of exposed services.
Quarterly recovery checkpoint
At least quarterly, perform a restore test using a copy of the backup. A backup that has never been restored is an assumption, not a recovery plan. Verify that you can recover configuration, mail data, credentials or secrets as designed, and DNS documentation. Record how long the test takes and which steps require manual intervention.
Mailbox migration plan
- Lower the old DNS record's TTL before the planned cutover, while recognizing that resolvers may cache records longer than expected.
- Recreate domains, mailboxes, aliases, quotas, and important filters in Mailcow.
- Use an IMAP migration method supported by your chosen tools or provider to copy historical mail. Preserve folders and verify message counts where practical.
- Publish and validate MX, SPF, DKIM, DMARC, and reverse DNS before changing the primary route.
- Change MX records, then keep the old mailbox available during the transition.
- Run an incremental copy after cutover to catch messages delivered during the transition.
- Update mail clients, mobile devices, applications, and contact forms to use the new submission hostname and credentials.
- Keep a written rollback plan and postpone decommissioning the old provider until mail flow and recovery have been tested.
How to interpret changes
When delivery changes suddenly, separate the problem into DNS, authentication, connection, reputation, and content categories. The following troubleshooting tree keeps the investigation focused:
- Can the recipient find your server? Check the MX record, the mail hostname's A or AAAA records, and the VPS firewall. If inbound mail fails, start with DNS and port reachability.
- Can your server identify itself consistently? Compare the SMTP hostname, forward DNS, reverse DNS, certificate names, and HELO or EHLO identity. A mismatch is a reason to correct configuration before changing spam settings.
- Does authentication pass? Inspect the recipient's headers for SPF, DKIM, and DMARC results. Check for multiple SPF records, an incorrect DKIM selector, expired keys, or a From domain that does not align with authenticated domains.
- Is the message deferred or rejected? Read the complete SMTP response and Mailcow logs. A temporary deferral should be monitored for retries; a permanent rejection usually requires correcting the stated cause rather than resending unchanged messages.
- Are messages accepted but filtered? Compare content, sending patterns, links, attachments, and authentication across successful and unsuccessful tests. Do not treat placement in a recipient's spam folder as proof that DNS is wrong.
- Is the server itself unhealthy? Check disk space, queue size, container status, memory pressure, certificate validity, and recent configuration changes. Resource exhaustion can look like a delivery problem.
For DMARC, begin with a monitoring policy such as p=none while you identify legitimate senders and fix alignment. Only tighten enforcement after reports show that expected mail passes. If third-party services send on behalf of your domain, document their authentication path before changing the policy.
When to revisit
Revisit this setup monthly for operational checks and quarterly for recovery and security review. Return immediately when adding a domain or mailbox, changing the VPS provider or IP address, rotating DKIM keys, modifying DMARC policy, replacing a reverse proxy, or migrating an application that sends mail.
Refresh the deployment notes after a Mailcow release, operating-system change, Docker update, DNS-provider change, or change to email authentication standards. Recheck the VPS provider's port policy whenever you move servers. Review TLS guidance and reputation signals after unexplained delivery failures rather than relying on an old test result.
Your next action should be to create a small operations record containing the server hostname, IP address, DNS records, Mailcow version, mailbox inventory, backup location, monitoring URLs, and rollback steps. Schedule a recurring review of that record. If you cannot explain how a message is authenticated, where a backup is stored, or how a mailbox would be restored, the deployment is not finished yet.
Mail is only one part of a broader VPS stack. Keep its data and credentials separate from unrelated services, and document dependencies before adding more Docker applications. Our guides to planning a Docker Compose self-hosted app stack and evaluating a VPS for Docker apps can help with those surrounding decisions.