Open rates: 6 lessons from digging deeper and a Gmail problem

Despite all the warnings, most of us still take open/render rates as a major indication of the success or failure of an email and, particularly, an email subject line.

Take a look at the unique open rates for my last six newsletters:

Pretty much steady as she goes. No dramatic peaks or troughs, so keep up the good work, make another cup of tea and be happy.

And so ends our open rate analysis.

Let’s try again.

Segment out, for example, subscribers with a gmail.com address and compare their open rates with everyone else:

Yikes!

Three issues immediately spring to mind:

• open rates at Gmail are way below the list average. Why?
• the open rates for Dec 14th and February 8th were unusually low for Gmail users. Why?
• open rates dropped faster at Gmail from January 11th than the average and then make an unexpected recovery. Why?

Until we dug deeper into our open rates, we thought everything was pretty stable. Now we know better.

Let’s explore some plausible explanations for fluctuating open rates at this address domain.

Theory 1: Email is not reaching the inbox

Segmenting results by domain is a great way to spot delivery problems with a particular webmail service, as this post describes.

How can we tell if delivery issues are responsible for, say, the December 14th open rate dip?

Remember, email marketing software and ESPs tells you how much email was successfully sent, but not how much was quietly deleted by the receiving organization or diverted to a junk folder.

To discover how much email likely made it to the inbox, you can use seed lists or monitoring tools provided by deliverability services or other techniques outlined here. We can also see if clicks were affected, too (hold that thought).

Theory 2: Image blocking

Another reason for low open rates is that Gmail blocks images from displaying in emails unless certain conditions are satisfied. For example:

• the recipient elects to view them by clicking an appropriate link
• the recipient has sent at least two emails to that sender

Since an open is only recorded when a tracking image displays, image blocking will depress open rates. Equally, Gmail has no standard preview pane, so no accidental opens are triggered by scrolling through your inbox, either.

This seems a good explanation for the general open rate malaise, but not the unusual dips.

Let’s check on clicks.

If deliverability issues are a problem, then we would expect clicks from Gmail users to be lower than average, too. You can’t click on an email you don’t receive.

If clicks are not below average, this suggests image blocking is the chief culprit: people are getting the emails and clicking as normal, just not seeing the images in the newsletter that are used to record opens.

What percentage of all Gmail users on the list clicked at least once on an email?

Goodness.

Maybe we should have just stuck to our simple open rate review, closed our eyes and drank more of that tea.

Check the December 28th result.

The open rate at Gmail is way below average, but the clickthrough rate (equivalent to clicks per Gmail subscriber) is completely normal. Clearly image blocking is to blame for much of the below-average open rate results.

This illustrates two things.

First, a lower open rate is sometimes meaningless, especially (as with my newsletter) if images are not a big factor in your email’s ability to drive response

Second, if we just looked at opens, we’d get a completely false picture of how our emails actually perform. My newsletter is designed to drive visitors back to the website: clicks matter more than opens.

OK, but the click graph also tells us that image blocking isn’t the only issue.

On December 14th, for example, the below-average CTR suggests that the poor open rate for that day at Gmail was also down to a delivery problem. Not enough Gmail subscribers saw the email in their inbox.

What about January 11th and after?

Looks like we have deliverability issues again. But is that all?

After all, the open rate drops aren’t as bad as on December 14th, yet the gap between CTR for Gmail users and non-Gmail users is far greater. Is something else going on?

Theory 3: Gmail users are different

On January 12th, Google announced they were progressively switching all Gmail users over to a more secure https interface setting.

One consequence is that many innocent images in HTML email now trigger a security warning popup in Internet Explorer when viewed at Gmail. For the full story on this issue, see this post by robinteractive.

Some of those IE users would previously have elected to display images (triggering opens), but may respond to those security warnings by blocking these images. This is almost certainly contributing to lower open rates for the January 11th newsletter and onwards.

However, as we learnt from the December 28th result, lower open rates doesn’t always imply fewer clicks if image blocking, rather than poor delivery rates, is to blame. So we’d expect clicks to behave like the December 28th result.

Except in the January 11th emails and later, clicks are indeed affected: CTR is rising for the list as a whole, yet dropping for Gmail subscribers.

How can this be?

First, maybe delivery issues are indeed playing some role.

Second, I wonder if the security warnings are having a psychological affect on the reader, making them less likely to trust the sender and less likely to click on links in the sender’s emails.

The growing “click gap” after January 11th is consistent with gradual rollout of the feature.

Further evidence for this interpretation comes from click-to-open rates.

If deliverability is the problem, then the gap in CTO rates between Gmail users and non-users should remain steady.

If more image blocking is happening at Gmail, then CTO should rise for Gmail users relative to other subscribers, as fewer opens are recorded for the same number of clicks.

But if the willingness to click is declining for some Gmail users through these security warnings, then the gap will change to favor non-Gmail users.

Click to open rates at Gmail

Here’s the graph. And it’s very revealing.

The difference between Gmail and non-Gmail recipients in terms of CTO is the same for December 14th and December 28th. That fits our theory that delivery problems hurt clicks December 14th.

Now look at the January and February numbers. It seems Gmail users are clicking less than they were before the switch to https, at least relative to non-Gmail users.

So the https switch seems to be having an impact on response.

What do we learn from all this?

All the above has several potential lessons and insights.

First and foremost, it’s clear that a quick look at your email open rates can produce completely misleading conclusions.

By taking a closer look at key segments of your list, and evaluating other, more important metrics, you can gain insights into potential problems and issues that will let you make the changes needed to improve results (assuming the interpretation was correct).

Specifically, by digging deeper into my numbers, I learnt:

1. I have delivery issues at a particular webmail service. I can now look into the particular needs of Gmail (or get outside help) to address those issues and improve deliverability.

2. I’ve discovered an important interface change at Gmail which is potentially hurting my email marketing. Fortunately there is something I can do about this, as robinteractive’s post explains.

3. It makes sense to track feature changes at major webmail services. Both Gmail and Yahoo! Mail, for example, have blogs that announce new functionality and other interface changes.

4. There are many reasons for open rate changes that are nothing to do with subject lines. It’s a mistake to always look at the subject line when open rates change. Here are some more factors that can affect these open rates.

5. Changes to open rates can sometimes be meaningless in practice.

6. Gmail users are generally regarded as more responsive than the average (see, for example, this MailChimp study), but that’s not reflected in my stats.

Remedial action might be to pay more attention to the snippet text that is featured in the Gmail inbox or to work harder to encourage people to submit their more valued work address, rather than generic webmail addresses (see the article on getting the right email address.)

Of course, all this is interpretation. But if nothing else it shows the value of taking some time to look harder and more holistically at those campaign reports.

P.S. Stat fans note that those graphs should be bars and not continuous lines, but continuous lines make it easier to demonstrate patterns.

Find related articles:

• Channel: Email analysis

You can follow any comments on this blog post through the RSS 2.0 feed.

Leave a comment