While building and running Newsberry, we have learned a lot about email delivery and getting messages to the inbox. The problem is, when I talk to most developers, the full picture of email delivery is not understood. I’d like to offer this article as a primer on email delivery, so developers can improve email deliverability in their web applications.
1. Choosing a mail server
This article will not go into the details of setting up a mail server, because I assume you are already sending email. All of the techniques for delivery can be achieved with most open-source and commercial mail servers. Depending on the volume of your email, you may want to choose a commercial SMTP server that has a focus on deliverability. We currently use Port25’s PowerMTA, which is extremely powerful and flexible when it comes to sending a large volume of email. Some other mail servers you can use include StrongMail, Qmail, Postfix, Sendmail, and Exchange 2003.
2. Setting up the domain and IP
This part is simple, but many times overlooked. I’ve seen some people just send directly from localhost or the default domain, which is not recommended. It is best to choose a separate IP address and sub-domain for your SMTP server. This way you can isolate your email architecture and setup a plan for scaling. There are a few simple things you should do first.
- Reserve at least one IP address for your SMTP server
- Choose a domain or sub-domain, such as mta1.domain.com
- Setup Reverse DNS for the IP that corresponds to the SMTP domain (Very important!)
The three things above are really simple, but the third is always overlooked. ISPs and mail servers will reject email without proper rDNS or send it to the bulk/junk folder. Make sure to contact your hosting provider to setup the proper rDNS entry. As you add SMTP instances, you can create them according to the first (mta2.domain.com, etc).
3. Email authentication
This is where things get a little tricky. At the moment, there are three main methods to authenticate your email. These standards are basically setup to prove that you can send for your domain. Ever get a spam email from yourself? With email authentication this would be impossible, because that sender would not be allowed to send on behalf of your domain.
The three standards are SenderID, SPF, and DomainKeys/DKIM. SenderID and SPF are basically the same, except for one important detail that I will get into below.
SenderID / SPF
SenderID and SPF are based on a TXT record that you add to the DNS of your domain, which lists the approved IP addresses that can send for your domain (See, it make sense to have a separate IP address for email). Let’s use wildbit.com as a quick example.
We currently use Google Apps for our email, but we also send email newsletters through Newsberry. So in our DNS for wildbit.com we have something like:
domain.com. IN TXT "v=spf1 a mx include:b.nbrmail.com include:aspmx.googlemail.com ?all"The line above says that email originating from wildbit.com can come from any A or MX Record, but we also include b.nbrmail.com and aspmx.googlemail.com as senders. The “include” method is unique because it will pull in the SPF records of that domain into my own, so this way any SMTP servers that Newsberry or Google uses are also approved for wildbit.com. If you are setting up SPF for the first time, your TXT record will look something like this:
domain.com. IN TXT "v=spf1 a mx ip4:123.123.1.34 ?all"The above states that the A and MX servers may send email, as well as the IP 123.123.1.34. This IP would be the one that you setup in step 2. If you have multiple SMTP servers, you can add the ip4 entry multiple times to list each one.
This might be a little confusing at first, but there are some simple wizards for SenderID and SPF that explain and generate this for you. The important thing is the ?all or -all at the end. The ? is sort of like a soft-approval, in case you might send from other domains that are not listed (webmail, comcast, etc). If you use the - you are saying that these are the only approved SMTP servers. If you know for sure, then I recommend the dash.
So, what is the difference between SPF and SenderID? The main difference is which header is verified in the sent email. Here is a snippet of the email header for an email sent from Newsberry, but setup with myself (cn@wildbit.com) as the sender.
Received: from m3.nbrmail.com (m3.nbrmail.com [206.196.26.231])
by mx.google.com with ESMTP id n29si7230906pyh.2007.09.04.11.36.13;
Tue, 04 Sep 2007 11:36:14 -0700 (PDT)
From: cn@wildbit.com
Return-Path: campaign_7456_287860@b.nbrmail.com
To: info@wildbit.comAs you can see, this email was sent from m3.nbrmail.com, which is one of Newsberry’s mail servers. Other than that we have From: and a Return-Path:. A Return-Path is used to send all automated bounces back to a specific address. The From is just what is says, who the email is from. This is where the difference comes in for SPF and SenderID. SPF will look at the DNS TXT record of b.nbrmail.com (Return-Path) to see if m3.nbrmail.com is allowed to send for it. SenderID will look at the DNS TXT record of wildbit.com (From) to see if m3.nbrmail.com is allowed to send for it.
Now, unless you need bounces to go to a specific address at a different domain, you don’t need to worry about this. If you do, this is very important to know, because you need to setup the TXT record for both domains.
Invite and Send-to-Friend Forms
The other scenario where this comes into play is with common “Invite” or “Send to Friend” tools. It is common for developers to send these emails from the person’s email address. This can cause problems, because your SMTP server is not approved to send email for that person. In this case you have three options:
- Add a Return-Path: header with an address from your domain. This will be validated by SPF, but not SenderID.
- Use a From: address from your domain, which will be validated by SPF and SenderID. In this case the person receiving the email might not recognize the email, but you could use the sender’s name.
- Add a Sender: header to the email. This will produce a valid SPF and SenderID email, but will show up as “on behalf of email@yourdomain.com” in the From area of the recipients email client.
DomainKeys / DKIM
DomainKeys is a standard that was pushed by Yahoo! and has become important for email delivery. The goal of DomainKeys is similar to SPF, to verify identity, but it also aims to create a profile based on that unique identity, keeping track of the sender’s reputation. DomainKeys are based on a public and private key pair. The public key is published in DNS as a TXT record and the private key is used to sign messages as they are sent.
DomainKeys are much more complicated to setup, because it involves setting up a key pair and it requires a mail server that can sign the messages. PowerMTA supports DomainKeys very well. You can take a look at the SourceForge site for DomainKeys, which provides specific instructions for most email servers.
The first step is to generate a key pair that will be used for outgoing messages. There is a simple tutorial on the DomainKeys site explaining how to do this. I also found a nice generator, but it is old and only supports a 384 bit key. We currently use 768, so maybe we will write a simple 768 key generator.
The next step is to insert this public key into DNS for the domain of the From: header. This is the tricky part (or at least it was for me). The format of the DNS entry is a little unusual, so much that some providers do not even support it. We use DNS Made Easy for our DNS management, but I know that GoDaddy also supports DomainKeys in their DNS. There are two entries that you need in the DNS record for your domain.
_domainkey.domain.com. IN TXT "t=y; o=~;"
m._domainkey.domain.com. IN TXT "k=rsa; t=y; p=MEwwDQYJKoZIhvcNAQEBBQADOwAwOAIxAMvolOXfnAZTVOk8YMhg5xtAZJ4V0n4KOwCpQVzj89iUxWoFmDmuqR0OnNy2gm5gTwIDAQAB"The first declares that you are using DomainKeys and the second defines the public key and selector. The selector is the m before ._domainkey in the second entry. From what I understand, this is mainly used to define multiple DomainKeys for one domain. To keep things simple, we will just use m as the selector.
Now that you have your DNS setup, you can sign keys with your mail server. Each mail server is different, so I suggest that you refer to the documentation, otherwise this article will never end. Here are some links to get you started:
4. Confirm that everything works
The final step, of course, is to test that it all works. Port25 has setup an email address that you can send a test to, which then replies with the results. Just email check-auth@verifier.port25.com. If you have a Gmail account you can also check the “show original” option and look at the headers. It should display the SPF and DKIM information in the headers. The headers should look something like this:
Delivered-To: cn@wildbit.com
Return-Path: campaign_56_2860@b.nbrmail.com
Received: from m3.nbrmail.com (m3.nbrmail.com [206.196.26.231])
by mx.google.com with ESMTP id n29si7230906pyh.2007.09.04.11.36.13;
Tue, 04 Sep 2007 11:36:14 -0700 (PDT)
Received-SPF: pass (google.com: domain of campaign_56_2860@b.nbrmail.com designates 206.196.26.231 as permitted sender) client-ip=206.196.26.231;
Authentication-Results: mx.google.com; spf=pass smtp.mail=campaign_56_2860@b.nbrmail.com; dkim=pass (test mode) header.i=@wildbit.com
DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; q=dns; s=m; d=wildbit.com;
h=From:To:Date:Subject:MIME-Version:Content-Type:Message-ID;
b=ckS+QuTR0+i3A6qgC0kNaXDQtcbbItv5R/6NdjxikxeH0MbFiRB7hdnNS3ygx5oIUaG0WsI5JYh8
OxTVZZj/VrVUe1JdVmy/cxaS6DZKsmFbY39pJQ5ac7AA0fuO0ORd
DomainKey-Signature: a=rsa-sha1; c=nofws; q=dns; s=m; d=wildbit.com;
b=hSYfBl6o3RQnzc0TWhotWASXwRcBcMEUmlViMTmGi5KNNdeXAyf+Q3OdW7LokRXUk1Wcwh10Hb8x
RaKeUYu+rlW945YVRO3CTG9xW9RZPCcq8j2AtfSheDhDwA+uARQS;
Received: from DED1413WILD (216.128.27.6) by m3.nbrmail.com id hrmipq0e66cf for <cn @wildbit.com>; Tue, 4 Sep 2007 14:36:13 -0400 (envelope-from campaign_56_2860 @b.nbrmail.com)
From: cn@wildbit.com
To: info@wildbit.com
Reply-To: cn@wildbit.com
Date: Tue, 04 Sep 2007 14:36:13 -0400Next steps...
I didn’t think that the technical side of things would be such a long post. I decided to split this into two parts. The second part of this article will explain the basics of white-listing, accreditation services, and monitoring (and protecting!) your IP addresses.
If you have any additions to this, please feel free to let me know. I’m always learning something new about email delivery. Also, feel free to ask any questions.