Improving Email Deliverability (Part 1 – Technical Checklist)

At Crafted, we create and manage many sites that send emails to users for a variety of reasons. For example, when a customer buys from a website, the system can be set up to send an email confirming their purchase.

Making sure that these emails end up in users' inboxes, instead of being marked as spam, bouncing, being rejected, or, in a worst case scenario, resulting in the site being blacklisted by ISPs, depends on a number of technical and content-based considerations.

As everyone with an email account knows, email spam is a very real problem; according to the Message Anti-Abuse Working Group, between 88–92% of email messages sent in the first half of 2010 were spam.

With that in mind, it’s natural and essential for email providers to screen emails received in order to filter out spam. As senders of legitimate transactional emails, it’s also our responsibility to make sure that our emails adhere to industry standards to enable email providers, such as Microsoft, Yahoo and Google, to more easily sort the junk emails from those that have a right to be in inboxes.

Unfortunately, only email providers know their spam filter algorithms, and they like to keep them under wraps. While there are a number of variables that we do know about, no email is ever guaranteed to arrive in someone's inbox.

So, how can you increase the chances of your emails making it to your intended recipients?

MX Blacklist

First of all, it's important to check that the IP Address used for sending email is not blacklisted. One great tool to do this is MX Tool Box.

Note that email is always sent from the server's primary IP address - having an additional static IP address may be listed in the mail headers, but the server's primary IP address will always be mentioned. So, be sure to check both IP addresses.

PTR Records / Reverse DNS Lookup

When a server sends an email to another, it sends a HELO message. This HELO message is checked against the PTR record of the IP address sending email. So if you’re sending from xxx.xxx.xxx.xxx, you should set up an appropriate PTR record, such as example.com or mail.example.com, and make sure that your mail server sends "example.com" / "mail.example.com" (as appropriate) as the HELO message.

SPF Record

SPF (Sender Policy Framework) is a system where the IP address sending the email is checked against a list of servers mentioned in the DNS records of the domain listed in the FROM header. DNS records are public, but can only be changed by the domain owner, proving that the email has been sent from an authorized server.

So, if you’re sending emails from confirm@example.com, the mail server receiving the email will check that the IP address is listed in the SPF record of example.com.

An SPF record is created as a TXT record with quite a specific syntax. Open SPF has details on the syntax required.

An example SPF record will look like this: v=spf1 a mx ip4:xxx.xxx.xxx.xxx ~all

You can specify the course of action to take in the final part of the SPF record. The different options are:

• -all : (fail) reject all emails that have been sent by a server not listed
• ~all : (softfail) treat emails that have been sent by a server not listed with caution
• other options include +all and ?all but these won’t help as they offer no instruction to the receiving mail server.

SenderID

SenderID is a technology built by Microsoft and is similar to SPF. Both technologies share a similar syntax and, in the absence of a SenderID record, the SPF record is treated as a SenderID record by servers implementing SenderID checks (Hotmail for example).

While both share a similar syntax, they differ in the internal implementation of the technologies. However, from a mail sending point of view, I’ve found that a valid SPF record will be fine. You’ll need to be careful though as I’ve also found that SPF is more tolerant than SenderID; including a domain in the SPF record that doesn’t have a SPF record will result in an error, whilst this is perfectly valid for email providers using SPF.

You can find out more about SenderID on Microsoft’s website:

DomainKeys / DKIM

DomainKeys and DKIM are similar technologies that authenticate an email’s content rather than just the FROM header. They use the concept of encrypted signature with a public key listed in the domain's DNS records.

Basically, the email message will include an additional header listing the DomainKey’s domain, the headers being signed and the encrypted signature.

Receiving mail servers use the public key to confirm that the message has been signed by the domain owner's private key. This is sometimes visible in certain email providers such as Gmail, which will display the "signed by" header.

Feedback loop (FBL)

Email providers such as AOL require a feedback loop system in place. This is a header in the email message that directs the receiving mail server to an email address or web service where a "spam complaint" can be made. Details about this can be found on AOL’s website.

In essence, implementing this technology shows that you address complaints made by users and servers about your emails. This isn’t only a positive factor for mail receiving servers, but may also help you to understand when and how email messages are being regarded as spam by either humans or servers.

Whitelisting

Major ISPs allow for mail sending companies to apply to be included in a whitelist. Each ISP will manage their own whitelist and have different requirements to be included. Public whitelists are also available.

IP Reputation

When sending email for the first time, it's important to remember that your IP address may not have any previous history of sending email. While this can be seen as a positive thing (it hasn't been used to send spam before), mail receiving servers don't have any information yet as to whether your server is good or bad, so it’ll treat your email with caution. This may cause delays and rate limiting.

It’s often recommended to ‘warm’ an IP address before using it intensively. By sending a smaller quantity of email, you’ll allow mail servers to get to know your server and determine that it’s good.

Content – A prelude to Part 2

One of the most important factors that ISPs use to determine whether an email is spam is its content.

Email content should usually be sent in both HTML and plain-text formats. HTML versions should include a reasonable amount of content, especially when including an image. This is because spam emails will sometimes include an image containing the spam content with little or no text recognisable by the mail server.

Email content should avoid certain phrases and words usually used in spam emails where possible and include the following pieces of information:

• an explanation of why the user is receiving the email
• a link / address that the user can use to unsubscribe themselves from receiving any future emails
• legal and contact information about the company sending the email, including the sender's physical location
• links to the Terms and Conditions and the Privacy Policy of the company sending the email.

This information is required in the United States by the CAN-SPAM Act.

Conclusion

There are a number of tools that you can use to reduce the likelihood of your emails being marked as spam, with port25.com reports being a good place to start. However, reading through the SMTP logs for any errors and looking over a received email's mail headers will give the best idea of why the email was rejected or marked as spam.

Different email providers use different techniques, so it's best to cater for all. You can find a breakdown of technologies used by email providers on the Mailchimp website.

It’s essential that you measure the deliverability of emails. You could invest in a bespoke solution to help you with this or use a third party SMTP service such as Sendgrid, which will provide vital statistics concerning emails.

At Crafted, we use Sendgrid to send transactional emails on behalf of our clients. The services provides PTR records, DomainKeys / DKIM, Feedback loops and whitelisting with major ISPs, as well as providing a great interface to measure the deliverability of emails.

By John Folley 

Digitally Crafted – a round-up of the week in digital marketing

In this week’s round up of the most important topics in digital marketing, we reveal Facebook’s giant takeover bid for Instagram, a revamp to Microsoft’s AdCenter and the valuable marketing lessons learnt from the 2012 Olympic Games.

Design and development

Getty Images bought by private equity firm Carlyle Group for £2.1bn
The world’s leading picture agency, Getty Images has been bought by private equity firm, Carlyle Group for a whopping £2.1bn. Founded in 1995, Getty Images has revolutionised the image industry, moving into online photography in a bid to erase doubts about the value of stock pictures. It has acquired several of its rivals in recent years, including UK picture firm Scoopt and online library, iStockPhoto. Read more

Paid search marketing

Microsoft revamps AdCenter for agency relationships
Microsoft has revamped its AdCenter for paid search users to enable advertisers to change or add agencies without losing their entire account history. In a move to enhance agency relations, the ability to link to existing client accounts will give agencies new to an account greater insight into its history, rather than starting from scratch. Read more

Social media

Twitter analytics company Followerwonk acquired by SEOmoz
Leading SEO community and tools provider, SEOmoz has announced a new addition to its product range, following the acquisition of Portland-based Twitter analytics brand Followerwonk. Followerwonk enables users to search for, compare analyse, track and sort followers to streamline Twitter usage. It is thought Followerwonk will continue to be rolled out as its own product under the SEOmoz umbrella. Read more

Facebook’s $1bn Instagram deal cleared by OFT
Facebook’s eye-watering $1bn acquisition of photo-sharing service Instagram has finally been given the green light by UK regulator, the Office of Fair Trading (OFT). The body launched a probe into the pending purchase in June to discover whether the merger of the two parties would prevent photos being uploaded to rival social media platforms such as Twitter. Read more

Facebook testing promoted news feed posts for non-fans
Leading brands may soon be able to advertise their products and services via news feeds of Facebook users who are NOT fans of their pages. The social network revealed earlier this week that it will begin to allow the testing of promoted posts that can reach users who haven’t “liked” a company’s Facebook page. An infringement of privacy? You decide. Read more

mCommerce

20% of mobile owners say mobile ads have influenced a purchase
A fifth of respondents in a survey by inMobi stated that they had been influenced to make an in-store purchase as a result of mobile advertising. More than two-thirds (63%) of UK mobile users have used mCommerce in some form according to the study, while 45% expect to do so within the next 12 months. Retailers are being encouraged to keep up with the trend to bind together a multi-channel shopping experience. Read more

Study: Mobile and video are key drivers of apparel purchases
A new study conducted by Google found a surprising and growing influence of mobile and video for apparel purchases. More than a fifth of apparel consumers used their tablets or smartphones on a daily basis for shopping, while 4 in 10 respondents had visited a store or retailer website as a result of viewing apparel video advertisements. Read more

Organic search marketing

July Search Share: Google at all-time high, Bing continues slow gains
The July search market share figures were formally released this week, with Google remaining at an all-time high position, despite not moving up on last month. Bing’s market share continues to grow slightly, while Yahoo’s market share growth also flatlined last month at 13 per cent. Interestingly, the combined Microsoft-Yahoo search alliance share stands at 28.7 per cent – 0.3 per cent down on the figure when the BingHoo alliance took effect. Read more

Google alters algorithm as Hollywood lobbyists win latest copyright battle
Google has made a significant change to its search algorithm this week, in a bid to devalue websites that persistently breach copyright laws. This move follows prolonged lobbying from Hollywood media and entertainment giants who have complained for years that the leading search engine does little to prevent access to material that breaches strict copyright laws on contact including music videos and television shows. Read more

Google no longer pushing Google+ results says search chief
Google search chief, Amit Singhal has revealed the search engine is no longer pushing Google+ results at the expense of other results, following criticism by Facebook and Twitter that it was unfairly promoting its own rival social network. In a recent interview with The Telegraph, Singhal admitted that all search results relating to Google+ have now “settled in a place which were better than when we (first) launched.” Read more 

Google: Further Penguin update “jolts” to come; Panda is smoother & monthly
Google has pre-warned spammers of its search engine that the anti-spam Penguin update is likely to receive a number of ‘jolts’ in the coming months in a bid to tighten its grip on black hat SEO. Meanwhile, updates to the Panda algorithm, designed to penalise low-quality web pages, will also be rolled out monthly. Google’s Matt Cutts has warned that sites will experience greater fluctuation due to Penguin ‘jolts’ as Google seeks to smooth out its newest algorithm. Read more

General marketing

Top 10 marketing lessons from the Olympic Games
For the marketing industry, London 2012 has been a wonderful opportunity to showcase the history and beauty of the capital. In the wake of a fortnight’s sporting competition, journalist Nicola Clark highlights the most useful marketing lessons from London’s historic Games – from emotion and eccentricity to brand engagement. Read more

adCenter developments - Microsoft finally introduces modified broad

Throughout the last couple of years, Google has released a myriad of updates, from the addition of Labels, enhancements to Ad Rotation and the addition of Auction Insights to name just a few.

One of the changes that particularly caught our eye was the addition of the Broad Match Modifier in late 2010. This allows advertisers to place plus signs in front of their keywords so that they can better control the types of searches that trigger their ads. Essentially, all keywords that are preceded by a plus sign must be included somewhere in a user’s search query.

This was a huge update from Google - it allows for greater reach than phrase match, but far more control than the basic broad match alone.

For example, with the keyword ‘colour contact lenses’ there were previously only three matching options:

coloured contact lenses – broad match

“coloured contact lenses” – phrase match

[coloured contact lenses] – exact match

Now with modified broad, advertisers can ensure that certain keywords are always present in a search query, whilst also taking advantage of the fact that these keywords don’t have to appear in a particular order (as with phrase match). They also no longer need to worry that their ads may be matched against entirely irrelevant searches, such a ‘camera lenses telephoto’ (as sometimes occurs with broad match).

We’re now able to run keywords in multiple ways, for example:

+coloured +contact +lenses

+coloured contact +lenses

+coloured +contact lenses

Our paid search team has seen some fantastic results from using modified broad. For example, over the last month, the modified broad keyword ‘+coloured +contact +lenses’ has seen a click through rate (CTR)  more than 5 times higher than the same keyword on broad match alone. It also generated a significantly higher sales volume.

This isn’t a one-off success either – we’ve seen similar improvements in click through rates and conversion rates across all of our accounts using modified broad match.

Microsoft recently decided to offer modified broad match as well. This is great news and is another indicator that AdCenter is continuing to move in-line with the service currently offered by Google.

In the coming month, Crafted will be looking to test the new modified broad match across all of our clients using Microsoft adCenter.

We are expecting to see some great results, so watch this space!

Written by John Stuchfield