5 lessons from a typical subject line test
Having no life to speak of, I’ve been thinking about subject lines.
Much of the advice I read is pretty absolute. Like:
“You dare not put everything in capitals.”
“Only Satanists use long subject lines.”
But there are few absolute truths in life, and even rules have exceptions.
Keep it short? Agree. Well, keep it as short as possible without sacrificing on potential impact. That’s not the same thing.
“We have clients where short and to-the-point subject lines…and others where lengthy subject lines tease you into reading the email and acting”
Any doubt about what works best FOR YOU is quickly dispelled with a simple A/B test. After all, you can’t argue with numbers, can you?
Unfortunately, that’s not an absolute truth either.
Test data is not hard to obtain. The tricky bit is the interpretation. And that’s where a lot of us go wrong.
So in the grand tradition of frizzy-haired 19th century scientists, I’m going to experiment on myself to reveal some of the problems you need to avoid.
What happens if you remove subject line branding?
Here’s a test I did with this site’s email newsletter.
The first 164 issues of that newsletter used a subject line that begins:
(Email Marketing Reports)…
So what happens if we remove that subject line branding? Place your bets now…
Here the results of an August A/B test (note that everything else about the email stayed the same):
Subject line 1: What YOU see versus what your readers see + how to get email attention
Open rate: 22.9%
Subject line 2: (Email Marketing Reports) What YOU see versus what your readers see + how to get email attention
Open rate: 21.8%
Given the size of the test samples, the small difference in open rate was not significant.
This echoes stats from MailerMailer, who aggregated results from emails with branded subject lines and compared them with those from campaigns without subject line branding.
They didn’t find much difference either.
Now let’s see where I messed up and dig out some issues with my rather typical subject line test.
Let’s begin with the obvious one.
The newsletter seeks to drive traffic back to the website. So “opens” is not actually a major goal, just a means to a click. Let’s look instead at click rates:
No branding: 16.6%
This difference is significant.
Removing the branding increased the number of clicks through to the articles at my site. I might also look at pageviews generated, unsubscribes, spam complaints and other metrics to get an overview of the true impacts.
|Lesson 1: Make sure you judge the test based on what’s important|
The trouble with using open rates as a judge is that subsequent metrics don’t always follow the same pattern. The winning subject line (based on clicks, sales etc.) does not always have the highest open rate.
It’s actually quite easy to write subject lines for high open rates. Segment by gender and send this to your male subscribers:
“Free beer as a thank you for subscribing”
You don’t want to do that?
Open rates are rarely your goal. And, thing is, you have to offer the free beer.
You don’t want subject lines that lift interest and expectations, only to shatter them with the actual content. That trains people to distrust your future subject lines. Which is “not good”.
Your short-term open rate boost may not produce the same lift in actual responses. And it may hurt future responses, too.
That’s why “urgency” works best when used genuinely and intermittently. If every few days you send out a “30% off for two days only”, the value of that urgency is lost to anyone who has received more than a handful of those messages.
And that’s why subject line “tricks” need careful and intelligent application.
How many changes are you actually testing?
Here’s the next problem: was this test comparing a branded with a non-branded subject line?
By removing the branding, the subject line also gets shorter.
So there are actually two changes at play here: branding and length. So what do we attribute the click benefits to?
Did the absence of branding get more people to take a second look to see what the email was about, rather than dismiss it as “another email from that marketing site”.
Or did the shorter subject line mean the actual information on the email’s contents was seen by more people?
Or was it a combination of both?
|Lesson 2: For future insight, ensure you understand how the subject lines you test actually differ, so you can attribute differences to specific characteristics|
The ideal subject line test tells you something you can apply in the future. But many tests look like this:
“30% off new summer fashions” vs “SUMMER SALE 2 Days Only!”
The results reveal a winning subject line for a particular campaign, which is a legitimate and useful outcome. But they give little insight on what it was that made that subject line better.
If subject 2 wins, was it the capitalization, length, urgency, etc. that made the difference?
Without such insights, you have few lessons to apply to future emails.
Net changes disguise negative impacts
And there’s another big danger when more than one thing changes…
Suppose the removal of branding actually hurts results, but the shorter subject line more than compensates for it?
If I then conclude that non-branded subject lines work better, I’m missing out on an important insight: that branding actually helps. Perhaps the best subject is something like this:
(EMR) What YOU see versus what your readers see + how to get email attention
Now let’s make it even more complicated.
Different people may be responding in different ways:
- The presence of a branded subject line may boost results for some who like my emails: they recognize the email and open it
- It may hurt short-term results for some who like my emails: they recognize it and file it for later review (better long-term results)
- It may hurt results for some who are ambivalent about my emails: they recognize it and know there’s no special reason to open it
- It may have no effect on readers using a smartphone, who might focus much more on the from line anyway
|Lesson 3: Try testing on different subscriber segments|
Removing branding will see response move in different directions, depending on the recipient. We only see the net effect.
My recent thoughts on reactivation campaigns reflect the fact that subject line elements may have different impacts, depending on the recipient segment.
Novelty versus real effects
If my kids get green pasta, they will show more interest in the meal.
Is it because green pasta is inherently more interesting than any other color pasta? Or is it because it’s different to the yellow pasta they normally get?
When you deviate from a regular pattern or approach in your subject line, there are two elements driving response changes.
The first is the impact of change itself: the novelty value.
The second is the fundamental impact of the change: what we might call the true impact.
Both can wear off or change with time, especially the first.
|Lesson 4: Be aware of the novelty factor when evaluating results and use it to your advantage|
The novelty factor in email marketing means changes like removing the branding or all capitals need reevaluation in a later test to see if the results still hold. Are the results repeatable?
Tests need repeating at intervals anyway – especially if your audience, content, goals etc. have changed and you need to revisit assumptions about what works best.
And, of course, the short-term attention boost that novelty value can sometimes get you is a legitimate tactic in its own right.
Chad White, for example, talks about “wake-up slap” tactics in email design, where deviation from the usual grabs attention. A similar concept might apply to subject lines.
The wider environment
So the results of any subject line tests also depend on that context.
I removed the branding from the subject. But…the from line is “Email Marketing Reports” and the preview pane features the site logo and the words ‘”The Email Marketing Reports newsletter”.
So the overall reduction in “branding” and “recognition” is considerably less than if the from line and preview pane didn’t also feature branding.
In the latter scenario, removing the branding might have hurt responses or lifted spam complaints as people had no clue who the email was from.
|Lesson 5: Test results may not be transferable between different email situations|
OK, let’s stop there as you simply must not write more than 500-700 words in an online article if you want a response (another one of those absolute truths).
Love to hear any of your views on how to better do and interpret subject line tests…comment below or on Google+
Find related articles:
You can follow any comments on this blog post through the RSS 2.0 feed.
12 comments on “5 lessons from a typical subject line test”