At Crafted, we create and manage many sites that send emails to users for a variety of reasons. For example, when a customer buys from a website, the system can be set up to send an email confirming their purchase.
Making sure that these emails end up in users' inboxes, instead of being marked as spam, bouncing, being rejected, or, in a worst case scenario, resulting in the site being blacklisted by ISPs, depends on a number of technical and content-based considerations.
As everyone with an email account knows, email spam is a very real problem; according to the Message Anti-Abuse Working Group, between 88–92% of email messages sent in the first half of 2010 were spam.
With that in mind, it’s natural and essential for email providers to screen emails received in order to filter out spam. As senders of legitimate transactional emails, it’s also our responsibility to make sure that our emails adhere to industry standards to enable email providers, such as Microsoft, Yahoo and Google, to more easily sort the junk emails from those that have a right to be in inboxes.
Unfortunately, only email providers know their spam filter algorithms, and they like to keep them under wraps. While there are a number of variables that we do know about, no email is ever guaranteed to arrive in someone's inbox.
So, how can you increase the chances of your emails making it to your intended recipients?
First of all, it's important to check that the IP Address used for sending email is not blacklisted. One great tool to do this is MX Tool Box.
Note that email is always sent from the server's primary IP address - having an additional static IP address may be listed in the mail headers, but the server's primary IP address will always be mentioned. So, be sure to check both IP addresses.
PTR Records / Reverse DNS Lookup
When a server sends an email to another, it sends a HELO message. This HELO message is checked against the PTR record of the IP address sending email. So if you’re sending from xxx.xxx.xxx.xxx, you should set up an appropriate PTR record, such as example.com or mail.example.com, and make sure that your mail server sends "example.com" / "mail.example.com" (as appropriate) as the HELO message.
SPF (Sender Policy Framework) is a system where the IP address sending the email is checked against a list of servers mentioned in the DNS records of the domain listed in the FROM header. DNS records are public, but can only be changed by the domain owner, proving that the email has been sent from an authorized server.
So, if you’re sending emails from firstname.lastname@example.org, the mail server receiving the email will check that the IP address is listed in the SPF record of example.com.
An SPF record is created as a TXT record with quite a specific syntax. Open SPF has details on the syntax required.
An example SPF record will look like this: v=spf1 a mx ip4:xxx.xxx.xxx.xxx ~all
You can specify the course of action to take in the final part of the SPF record. The different options are:
- -all : (fail) reject all emails that have been sent by a server not listed
- ~all : (softfail) treat emails that have been sent by a server not listed with caution
- other options include +all and ?all but these won’t help as they offer no instruction to the receiving mail server.
SenderID is a technology built by Microsoft and is similar to SPF. Both technologies share a similar syntax and, in the absence of a SenderID record, the SPF record is treated as a SenderID record by servers implementing SenderID checks (Hotmail for example).
While both share a similar syntax, they differ in the internal implementation of the technologies. However, from a mail sending point of view, I’ve found that a valid SPF record will be fine. You’ll need to be careful though as I’ve also found that SPF is more tolerant than SenderID; including a domain in the SPF record that doesn’t have a SPF record will result in an error, whilst this is perfectly valid for email providers using SPF.
You can find out more about SenderID on Microsoft’s website:
DomainKeys / DKIM
DomainKeys and DKIM are similar technologies that authenticate an email’s content rather than just the FROM header. They use the concept of encrypted signature with a public key listed in the domain's DNS records.
Basically, the email message will include an additional header listing the DomainKey’s domain, the headers being signed and the encrypted signature.
Receiving mail servers use the public key to confirm that the message has been signed by the domain owner's private key. This is sometimes visible in certain email providers such as Gmail, which will display the "signed by" header.
Feedback loop (FBL)
Email providers such as AOL require a feedback loop system in place. This is a header in the email message that directs the receiving mail server to an email address or web service where a "spam complaint" can be made. Details about this can be found on AOL’s website.
In essence, implementing this technology shows that you address complaints made by users and servers about your emails. This isn’t only a positive factor for mail receiving servers, but may also help you to understand when and how email messages are being regarded as spam by either humans or servers.
Major ISPs allow for mail sending companies to apply to be included in a whitelist. Each ISP will manage their own whitelist and have different requirements to be included. Public whitelists are also available.
When sending email for the first time, it's important to remember that your IP address may not have any previous history of sending email. While this can be seen as a positive thing (it hasn't been used to send spam before), mail receiving servers don't have any information yet as to whether your server is good or bad, so it’ll treat your email with caution. This may cause delays and rate limiting.
It’s often recommended to ‘warm’ an IP address before using it intensively. By sending a smaller quantity of email, you’ll allow mail servers to get to know your server and determine that it’s good.
Content – A prelude to Part 2
One of the most important factors that ISPs use to determine whether an email is spam is its content.
Email content should usually be sent in both HTML and plain-text formats. HTML versions should include a reasonable amount of content, especially when including an image. This is because spam emails will sometimes include an image containing the spam content with little or no text recognisable by the mail server.
Email content should avoid certain phrases and words usually used in spam emails where possible and include the following pieces of information:
- an explanation of why the user is receiving the email
- a link / address that the user can use to unsubscribe themselves from receiving any future emails
- legal and contact information about the company sending the email, including the sender's physical location
This information is required in the United States by the CAN-SPAM Act.
There are a number of tools that you can use to reduce the likelihood of your emails being marked as spam, with port25.com reports being a good place to start. However, reading through the SMTP logs for any errors and looking over a received email's mail headers will give the best idea of why the email was rejected or marked as spam.
Different email providers use different techniques, so it's best to cater for all. You can find a breakdown of technologies used by email providers on the Mailchimp website.
It’s essential that you measure the deliverability of emails. You could invest in a bespoke solution to help you with this or use a third party SMTP service such as Sendgrid, which will provide vital statistics concerning emails.
At Crafted, we use Sendgrid to send transactional emails on behalf of our clients. The services provides PTR records, DomainKeys / DKIM, Feedback loops and whitelisting with major ISPs, as well as providing a great interface to measure the deliverability of emails.
By John Folley