Anyone who knows me knows how cheap I am, especially when it comes to paying for services I feel I can run myself. Email is no exception. I've used Gmail since it was invite-only beta, but I've always wanted to have email that was untethered from the big email hosting companies out there. In the past I attempted to run a mail server through a terrible VPS provider and though I got it all working, I was burned when the VPS hosting company suddenly went under in the middle of the night.

This brings us to the present. When I started this blog, I wanted to host my own email in some fashion without just forwarding my domain specific email to a Gmail address. In this post we are going to go over a setup I put together over the course of a few hours that allows me to host mail in the cloud nearly for free.

Overview

This setup is surprisingly simple yet very flexible. To get this up and running you'll need to set up the following:

The Linux server in this setup can be hosted in the cloud or a home server. Most ISPs don't block the ports required to run an IMAP server at home (you could also use a non-standard port). Many ISPs also have extremely long leases on IP addresses. If you want something more stable though, a cloud hosted instance might be a better option. That being said, AWS does provide a free-tier EC2 option that should be able to handle a Dovecot install.

AWS SES

AWS SES is designed for programatically sending marketing blasts and transactional email. There are a few interesting features that make it work well for personal email as well. Setting up SES is pretty simple:

At this point, your domain is ready to send email. In the next section we'll go through what needs to happen to set up your first email address.

Before we get an email account set up, we also need to configure SMTP via SES. This is what allows you to send mail from your domain via an email client. To get this configured:

At the end, it will give you the details you need to connect your mail client to SES.

TIP For further compatibility, I would recommend setting up a custom MAIL FROM domain. You can configure this setting via the Domains option from the SES Dashboard. In my case I set up a sub-domain of mail and used it as my FROM domain. When you configure this it will give you a new MX record to add to your domain as well as an SPF record that is used by email servers to validate that your email came from the correct domain.

AWS SES to S3

One option you have with SES is to send all incoming mail to a specific S3 bucket. This is a super flexible option. Once the files are in S3, you can process them any way you see fit. In my bucket I have a processed directory where I move mail that I have already processed on my local IMAP server.

Check out the AWS Documentation on setting up SES to forward to an S3 bucket here.

TIP Set up a lifecycle policy on your S3 bucket to remove files older than 30 days. This will keep costs down and will also speed up mail retrieval by the Python script later in this post.

Dovecot

Next, we will set up an IMAP server to serve our mail for consumption by an email client. This is pretty dead simple. I do recommend getting a TLS cert for your domain. The best way to do this is to use the DNS verification method of the Let's Encrypt cert generation process. You can read more about that here. In order for this cert to work you'll also need to make sure to set up a DNS A record for your home server. I personally created an A Record to point the imap subdomain to my home IP address and created the cert for that subdomain as well.

I am not going to cover how to install Dovecot — it should be in your distribution's package manager — but I will provide my configuration file:

mail_home=/var/mail/%n
mail_location = mbox:~/mail:INBOX=/var/mail/%u

# if you want to use system users
passdb {
driver = pam
}

userdb {
driver = passwd
args = blocking=no
}

ssl=yes
ssl_cert=</etc/dovecot/ssl/cert.pem
ssl_key=</etc/dovecot/ssl/privkey.pem
# if you are using v2.3.0-v2.3.2.1 (or want to support non-ECC DH algorithms)
# since v2.3.3 this setting has been made optional.
#ssl_dh=</path/to/dh.pem

namespace {
inbox = yes
separator = /
}

protocols = imap 
disable_plaintext_auth = yes
mail_privileged_group=mail

This configuration will give you a functional IMAP server that uses your local username/password for authentication. For example, if you have a local account named "mark" you would authenticate with your IMAP server with the same username and password the "mark" user would use to log in to a shell on that server.

This user does NOT need to match the email address you want to use with SES. In the next step, we will be pull email out of S3 and put it in the local server's mailbox for a user of our choice (via the /var/mail/<user> mbox file).

A Little Python

At this point we have incoming mail hitting an S3 bucket and a functional local IMAP server; now what? With a little Python code, we are going to download these mail message files to our local server and parse them into a standard Linux MBOX file. The MBOX format is simple: messages are stored in one big file, with new messages appended to the end. You can read more about the MBOX format here.

I have created a Python script that does everything I need to get these messages downloaded to my local server that does the following:

The MBOX message delimiter line is pretty straightforward. It must start with the word From, followed by the sender name, email address, and the utc date/time the message was received in Unix ctime format (for example: Fri 07 Feb 2020 20:24:40)

The script is fairly simple:

import sys
import boto3
from datetime import datetime
from email.parser import Parser
from email.policy import default

mail_box = '/var/mail/<user>'
msg_dir = '/tmp/'
s3_bucket = '<s3-bucket-name>'

# Create s3 client
client = boto3.client('s3')

# Get new mail
response = client.list_objects_v2(
    Bucket=s3_bucket,
    Delimiter=','
)

new_messages = [msg['Key'] for msg in response['Contents'] if not msg['Key'].startswith('processed/')]

if new_messages:
    s3 = boto3.resource('s3')
    for i in new_messages:
        try:
            message_content = s3.Object(s3_bucket, i).get()['Body'].read().decode()
        except UnicodeDecodeError:
            message_content = s3.Object(s3_bucket, i).get()['Body'].read().decode('cp1252')

    # If the e-mail headers are in a file, uncomment these two lines:
        try:
            content = Parser(policy=default).parsestr(message_content)

            # Or for parsing headers in a string (this is an uncommon operation), use:
            with open(mail_box, 'a') as mb:
                if 'Date' in content:
                    header_date = content['Date']
                else:
                    header_date = datetime.utcnow().strftime("%a %d %b %Y %H:%M:%S +0000")

                from_addr = content['From'].replace('"','')
                header_string = f"From \"{from_addr}\" {header_date}"
                mb.write(header_string)
                mb.write(message_content)

            s3.Object(s3_bucket,f"processed/{i}").copy_from(CopySource=f"{s3_bucket}/{i}")
            s3.Object(s3_bucket,i).delete()
        except:
            print("Something went wrong!")
            print(sys.exc_info()[0])

I will admit this could be cleaned up a bit, but it's a v1 project. Fill in the mail_box and s3_bucket variables with the location of your MBOX file and the s3 bucket you created for SES. After that, you can schedule this to run in a cron job every minute and you should start seeing emails come through to your mail client.

DMARC

While not required, setting DMARC is a good way to make sure your mail is getting delivered and accepted by most mail servers. To set up DMARC, add a new DNS TXT record with a name of _dmarc and a value of "v=DMARC1;p=quarantine;pct=25;ruf=mailto:<email address>" replacing the email address with your new address. This will send you reports if your outgoing email is ever blocked by a destination mail server.

Conclusion

With this setup, when someone sends mail to your address, AWS SES receives it and stores the message in an S3 bucket. A Python script then pulls the message down from S3 and appends it to the local MBOX file. Finally, that message is served up via a local IMAP server. It looks complicated but if you have experience with Linux and AWS, it shouldn't be too difficult to set up.

I have been running this setup for over a year now and it's been working great. I had a few issues early on (and still find an occasional email that messes things up) when I discovered that not all incoming message headers are formatted consistently, but luckily I found out about the Python email module that handles this for me. I like that this gets me away from the big mail providers (I am not ignorant to the fact that Amazon is likely scanning all of my incoming mail) and that I am free to do what I want with the message files (like storing them off site, or putting them into a document store for better searching). While it's not for everyone, I am impressed with how much you can get done only using free tier services.