Last updated on March 15th, 2018 at 07:53 pm
E-mail is old and complex. It’s the oldest still-recognizable component of the Internet, with its modern incarnation having coalesced out of several different decades-old messaging technologies including ARPANET node-to-node messaging in the early 1970s. And though it remains a cornerstone of the Internet—the original killer app, really—it’s also extraordinarily hard to do right.
We most often interact with e-mail servers through friendly Web-based front-ends or applications, but a tremendous amount of work goes into hiding the complexity that allows the whole system to work. E-mail functions in a poisoned and hostile environment, flooded by viruses and spam. The seemingly simple exchange of text-based messages operates under complex rules with complex tools, all necessary to keep the poison out and the system functioning and useful in spite of the abuse it’s constantly under.
From a normal person’s perspective, e-mail seems like a solved problem: sign up for Internet access and your ISP gives you an e-mail address. Google, Apple, Yahoo, or any number of other free e-mail providers will hook you up with e-mail accounts with gigabytes of space and plenty of cool value-added features. Why do battle with arcane dragons to roll your own e-mail solution?
I’ll tell you why: because if it’s in the cloud, it’s not yours.
Because you must rely on others for your security. You have no control over who can read your correspondence—you must allow your data to be mined and your marketing profile extracted. You won’t be told if your metadata is collected or if your inbox is vacuumed up by a secret government request. You consent to be not a customer but a product, and a product has no rights.
Well, to hell with that. It’s your e-mail. And we’re going to take it back.
This is hard and even a bit scary…
E-mail is hard. If you want an easier sysadmin project, go set up a Web server. E-mail is a lot more complex, with many more moving parts. On the other hand, your correspondence with others is one of the most personal aspects of your online life—in a medium ultimately made of text, your words are you. It’s worth learning how to claw your online life back from those who would data mine and monetize it.
There are pitfalls and caveats—the biggest of which is that if you run your own e-mail server, you will be the sysadmin. The upside of this is that no bored or tired customer service rep about to go off-shift is going to fall for a social engineering attack and reset your e-mail password. The downside is that you are responsible for the care and feeding of your system. This is not an impossible task—it’s not even really difficult—but it is non-trivial and never-ending. Applying critical updates is your responsibility. When do critical updates come out? That’s your responsibility to keep track of, too.
Worst of all, if you screw up and your server is compromised or used as spam relay, your domain will almost certainly wind up on blacklists. Your ability to send and receive e-mail will be diminished or perhaps even eliminated altogether. And totally scrubbing yourself from the multitude of e-mail blacklists is about as difficult as trying to get off of the TSA’s No Fly list.
You have been warned.
…but it’s also worth doing
OK, that ought to be enough to scare away the people who aren’t serious. For those of you still with me: this is going to be a hell of a lot of fun, and you’re going to learn a lot.
This is going to be multi-part series, and here in this first part we’re going to ask (and answer) a bunch of questions about how we’re going to set our e-mail server up. We’ll also outline the applications we’re going to use and talk about what they do. We expect this series will run over the course of the next few weeks; unlike our series on setting up a Web server, though, you won’t be able to get started firing off e-mails after part 1—you need the whole thing in order for it all to work right.
This certainly isn’t the only DIY e-mail tutorial on the Web. If you’re eager to skip ahead and get started now, we suggest consulting Christoph Hass’ excellent tutorial on Workaround.org—he makes many (but nowhere near all) of the same configuration choices that we will be making. However, Ars wouldn’t be putting this guide together if we didn’t have a few tricks up our sleeves—we’ve been in an e-mail configuration cave for the past month, and we have a lot of good information to share.
Prerequisites and assumptions—the where and the how
So you want your own e-mail server. Excellent! The first decision, before we even get into things like operating systems and applications, is where you’re going to put it. If you’re on a residential ISP connection, you will face a number of challenges in running an e-mail server out of your closet. In addition to almost certainly finding the standard set of e-mail TCP ports blocked, your IP address is also almost certainly already on one or more blacklists in order to cut down on the amount of spam being spewed out by virus-infected home computers. Whether or not you’re actually spewing any spam is irrelevant—that ship has long since sailed, and residential IP addresses are almost universally considered poisoned. There are numerous tools you can use to see if your address is on a blacklist—make sure to check before you start.
If you just want to mostly follow along at home with a non-functional test domain for learning, then a virtual machine or spare closet server will do just fine; if you want to do it for real, you’ll either need to be on a business-class connection with unblocked ports and a non-blacklisted IP address, or you’ll need a hosting service. You don’t need a monster dedicated server or anything, but you do need at least a VPS you can install software on from the command line. There are many options; I always recommend A Small Orange or Lithium Hosting, but if you’re willing to sacrifice some performance, you can almost certainly host a small e-mail server on a free Amazon EC2 instance.
You’re also going to need a domain (again, unless you’re going to just play along and use a nonexistent test domain), and that means you’re going to need a registrar and an external DNS provider. My personal recommendations for registrars are Namecheap and Gandi.net; both took hard anti-SOPA stances (see these ) and both offer two-factor authentication options. I have used both registrars, and they are both excellent.
One of the lessons reinforced by the recent @N Twitter account theft is that you should segregate your online services where it makes sense to do so. A significant component of the @N compromise came from the attacker gaining access to Naoki Hiroshima’s GoDaddy account, with GoDaddy functioning not only as his registrar but also as the authoritative DNS source for Hiroshima’s domains. Once in, the attacker was able to change at least one of those domains’ MX records and thereby hijack delivery of that domain’s e-mail.
We’re going to attempt to mitigate that specific risk by using a separate DNS provider—specifically, we’re going to use Amazon’s Route 53 DNS service. That will limit the amount of immediate damage an attacker can do in the unlikely event of a compromise at your registrar.
“Ah,” you say, “but if I use Amazon EC2 for my e-mail server and Amazon Route 53 for DNS, then I’m not segregating at all!” This is true, but Amazon gives you rich access control between different services; it’s not difficult to ensure that one set of login credentials can only modify your EC2 server and a different set of credentials can only modify your Route 53 DNS settings.
There are also many other DNS providers if you want to physically distribute your eggs rather than rely on access control—and being paranoid about security is never unwise. For this guide, though, we’ll be walking through the specific steps that I took when taking my own existing Google Apps-hosted domain and e-mail private—that means a physical server and Route 53 DNS (which ends up costing me about $2 a month).
The who and the what
It’s best to have a plan for your e-mail when diving into this setup process: are you going to be starting fresh with your own domain and not migrating any older e-mail inboxes into it? Or are you going to take an inbox and folders from somewhere else and sync or copy to your new server? Once we get rolling, we’ll walk through that second scenario—taking an existing e-mail address on a custom domain and migrating it directly to your own server. However, the steps will work just fine for moving from, say, a Gmail address or an ISP-provided address.
Now we finally come to the “what” part: what operating system and what software. Strap in, because first we need to do a quick crash course in e-mail terminology.
MTAs, MDAs, and MUAs
The applications that send, receive, and deliver e-mails are categorized by their role in the process, and those roles are abbreviated by a series of three-letter acronyms. Most people are familiar with their MUA—that’s a Mail User Agent, more commonly called an “e-mail client” or an “e-mail program.” An MUA is a program like Outlook or Thunderbird—it’s the thing you run on your computer that you send and receive e-mail with. E-mail is a standardized tool, and you can generally use whatever MUA makes you happy.
At the apex is the MTA, or Mail Transfer Agent. This is the core application that actually transmits e-mail around between servers—applications like Exim, sendmail, Postfix, and qmail. We’re going to be setting up Postfix as our MTA, which will give us a good balance of flexibility, interoperability, and power.
Sandwiched in the middle between the MUA on your desktop and the MTA on the server is another application category: the MDA, or Mail Delivery Agent. The MDA gets messages from the mail server into the users’ inboxes, most commonly with the POP or IMAP mail protocols. Except under some very limited circumstances, an MTA isn’t going to do you much good without an MDA, so we’re going to be using Dovecot as our MDA.
There are other MxA categories too, but Postfix and Dovecot go together like peas and carrots. Between the two of them, we’ll be set from a mail transmission and delivery perspective.
The operating system: Linux
Our choice of tools dictates our choice of operating systems: we’re going to be running this on Linux—specifically, Ubuntu Server 12.04 LTS, because for our purposes it’s the most universally accessible and configurable (and it’s also available on the Amazon EC2 free tier).
Ubuntu Server gives us quick access to everything we need—not just Postfix and Dovecot, but also all the supporting tools we need to get e-mail running securely.
Oh, yes—we’ll have a number of additional things to get running, because this is not a throwaway fast tutorial. We’re going to do this right, and that means by the time we’re done, we’re going to have set up and configured all of this:
- Postfix, to send and receive e-mail
- Dovecot, for IMAP
- SpamAssassin, to keep spam out of your inbox
- ClamAV, to filter out viruses
- Sieve, to set up mail filters and rules
- Roundcube, for webmail
- PostgreSQL (or MySQL/MariaDB), for Roundcube’s database
- Nginx and PHP-FPM, to serve out Roundcube over the Web
And that’s just the tip of the iceberg. On top of installing all of this stuff, we’ll be walking through hardening and security steps for each, especially Roundcube (to which we’ll be adding PKCS12-based certificate authentication and two-factor logons with Google Authenticator).
We’ll also be going through all the ancillary steps needed to make certain your e-mails are received correctly—making sure you have SSL/TLS certificates, setting up DKIM and SPF, ensuring your MX and reverse PTR records are correct, and a bunch of other stuff. And we’re even going to find a bit of time in here to make sure you can send and receive e-mails with PGP encryption.
We are, in summary, going to do a whole lot of stuff.
OK, now I’m scared and overwhelmed
Don’t be, because this is Ars Technica, and you’re smart. We’re going to break down each and every step and walk through just about every line in every config file as we go. By the end of this, you’re going to be like (TV’s, not wrestling’s) Steve Austin: better, stronger, faster.
We’re taking a very Unix-like approach to e-mail rather than a monolithic (Microsoft-like) one. Unix-like operating systems are generally made up of a number of discrete tools that are each very good at a very limited number of tasks; this gives you modularity and customizability but often at the expense of complexity. For e-mail, we’re going to bind together the above list of discrete applications into a fully armed and operational battle station.
There are, however, plenty of other ways to skin the e-mail cat. If you really are starting from scratch and just want a working Linux-based mail stack without going through the big setup, there’s iRedMail—it’s a prepackaged bundle of almost all of the same tools we’re going to set up, and you can get it going in minutes.
There’s also that monolithic Microsoft way, and you can (relatively) easily set up Microsoft Exchange Server if you’re more comfortable with the Microsoft ecosystem. There are a number of disadvantages, though, and chief among them is that Exchange isn’t free. If you want to play on that side of the fence, though, you’ll have to find another guide—we’re sticking with Postfix and friends.
The when: Right now! Buy a domain!
We’ll close out part 1 here by nailing down the two biggest prerequisites we need to get started: a domain and a server.
If you’ve never registered a domain before, it can look like a daunting process. However, all you need is a registrar and a credit card (and, of course, a name in mind). Pop over to, say, Gandi.net’s search form and poke around to see what’s available. Dangerrocket.com or vorpalsnake.com or attackweasel.com could each be yours for $15.50!
Find one you love and buy it for a year. Just remember that a domain that sounds hilarious might not make the best impression on a potential employer if you use it for e-mail… although “[email protected]” does actually sound pretty badass.
It is an ICANN requirement that domain registrars publicly list contact information for Internet domains (with different top-level domains having different rules for how and how much information must be displayed). Linux and OS X users can use the
whois command line utility (or anyone can use online tools like this one) to display that info for domains. For example, doing a Whois lookup on arstechnica.com shows you who owns it (Condé Nast Digital), as well as who the administrative and technical contacts are.
If you register a domain, the information you provide will be similarly visible—clearly not an ideal situation for a private individual who doesn’t want his or her personal information freely available. Most registrars offer a service variously called “WHOIS protection” or “Privacy Guard” or other similar names, where they substitute in their own information over yours. Some registrars charge an additional fee for the service, and some (like Gandi.net) offer it free.
Whatever registrar you choose, opt for their Whois protection service. It’s one thing for a business to have contact information visible like this; it’s quite another for individual users to be so exposed.
Two-factor that registrar
Whatever extra security measures your registrar of choice offers, you should enable them. Not taking advantage of extra security is like not taking advantage of an employer’s 401(k) matching—if they offer it and you don’t do it, you’re just plain dumb. Namecheap, for example, allows you to disable password resets, and it does two-factor authentication by texting codes to your mobile phone.
Gandi.net, on the other hand, gives you the option to restrict logging in to a certain IP address range. It offers two-factor authentication via a TOTP application like Google Authenticator.
Whatever your registrar has, turn on every bit of it. It will make logging in a little less convenient, but it will also make it that much more difficult for an attacker to wrest away control of your domain.
Stack dat app
If you’re installing Ubuntu Server from scratch, the only server component you need to select during installation is the OpenSSH server (which we’ll eventually switch over to key-based logon before we’re done with this series). Rather than installing Dovecot and Postfix separately, there’s a single package that will pull them both down and do the lion’s share of configuration to make the two applications plug into each other. Run the following command:
After a moment, the package will shunt you into a screen asking you to choose a basic configuration for Postfix. Out of the available options, we want “Internet site” because we’re going to (eventually) be sending and receiving our mail directly with Postfix rather than relaying it through another server.
The next configuration dialog asks you to put in your server’s domain name in order for the server to know what domain it’s primarily going to be sending and receiving e-mail for. If you bought a domain, stuff the name in there; if you’re just testing or don’t intend to make this server public, something like “local.loc” or another nonexistent local domain should be used.
After that, just let the installation happen. You’ll wind up with a nearly production-ready installation of Postfix and Dovecot.
Wait, that was easy!
Well, sure—but we’ve only just barely begun. In part 2, we’re going to dig deep into Postfix’s and Dovecot’s guts and wire the two of them together even tighter. We’ll grab an actual for-real SSL/TLS certificate because your mail server will need to be able to properly and securely identify itself to other mail servers. We’ll decide on where and how to store your mailbox account names and passwords, along with the mailboxes themselves. By the time we’re done with part 2, we’ll have darn near a fully functioning mail set-up—but we won’t be ready yet for prime time.
In part 3, we’ll bolt on the required accessories: antispam, antivirus, and server-side filtering. We’ll also get Roundcube up and running so you’ll have webmail. We’ll lock that webmail down tight, too.
Along the way, we’ll also be sprinkling in all the required extra bits to ensure your e-mail will be well-received by other servers: we’re going to set up DKIM (DomainKeys Identified Mail) and the required DNS records to make it work, along with SPF records for another layer of validation.
My goal—assuming nothing else crazy comes up in the meanwhile—is to run one piece per week, so if you’re following along at home, we’ll be done in a month. Even if it looks like it’s going to be daunting, trust me: this is a journey worth taking.