5 lessons from a typical subject line test


Latest posts | By Mark Brownlow | 12 Comments | Licence this content

a lessonHaving no life to speak of, I’ve been thinking about subject lines.

Much of the advice I read is pretty absolute. Like:

“You dare not put everything in capitals.”

Or:

“Only Satanists use long subject lines.”

But there are few absolute truths in life, and even rules have exceptions.

Shouldn’t use all capitals? Probably not, but it works for Overstock and the Tumbleweed Tiny House Company.

Keep it short? Agree. Well, keep it as short as possible without sacrificing on potential impact. That’s not the same thing.

See, for example, the work by Alchemy Worx on how longer subject lines can benefit click rates. As Simms Jenkins notes:

“We have clients where short and to-the-point subject lines…and others where lengthy subject lines tease you into reading the email and acting”

Any doubt about what works best FOR YOU is quickly dispelled with a simple A/B test. After all, you can’t argue with numbers, can you?

Unfortunately, that’s not an absolute truth either.

Test data is not hard to obtain. The tricky bit is the interpretation. And that’s where a lot of us go wrong.

So in the grand tradition of frizzy-haired 19th century scientists, I’m going to experiment on myself to reveal some of the problems you need to avoid.

What happens if you remove subject line branding?

Here’s a test I did with this site’s email newsletter.

The first 164 issues of that newsletter used a subject line that begins:

(Email Marketing Reports)…

The theory is that consistent subject line branding means people recognize the email in their inbox. Recognition is good (except when it’s bad).

So what happens if we remove that subject line branding? Place your bets now…

Here the results of an August A/B test (note that everything else about the email stayed the same):

Subject line 1: What YOU see versus what your readers see + how to get email attention
Open rate: 22.9%

Subject line 2: (Email Marketing Reports) What YOU see versus what your readers see + how to get email attention
Open rate: 21.8%

Given the size of the test samples, the small difference in open rate was not significant.

This echoes stats from MailerMailer, who aggregated results from emails with branded subject lines and compared them with those from campaigns without subject line branding.

They didn’t find much difference either.

But wait…

Now let’s see where I messed up and dig out some issues with my rather typical subject line test.

Open rates?

Let’s begin with the obvious one.

The newsletter seeks to drive traffic back to the website. So “opens” is not actually a major goal, just a means to a click. Let’s look instead at click rates:

Branding: 13.3%

No branding: 16.6%

This difference is significant.

Removing the branding increased the number of clicks through to the articles at my site. I might also look at pageviews generated, unsubscribes, spam complaints and other metrics to get an overview of the true impacts.

Lesson 1: Make sure you judge the test based on what’s important

The trouble with using open rates as a judge is that subsequent metrics don’t always follow the same pattern. The winning subject line (based on clicks, sales etc.) does not always have the highest open rate.

It’s actually quite easy to write subject lines for high open rates. Segment by gender and send this to your male subscribers:

“Free beer as a thank you for subscribing”

You don’t want to do that?

Quite.

Open rates are rarely your goal. And, thing is, you have to offer the free beer.

You don’t want subject lines that lift interest and expectations, only to shatter them with the actual content. That trains people to distrust your future subject lines. Which is “not good”.

Your short-term open rate boost may not produce the same lift in actual responses. And it may hurt future responses, too.

That’s why “urgency” works best when used genuinely and intermittently. If every few days you send out a “30% off for two days only”, the value of that urgency is lost to anyone who has received more than a handful of those messages.

And that’s why subject line “tricks” need careful and intelligent application.

How many changes are you actually testing?

Here’s the next problem: was this test comparing a branded with a non-branded subject line?

No.

By removing the branding, the subject line also gets shorter.

So there are actually two changes at play here: branding and length. So what do we attribute the click benefits to?

Did the absence of branding get more people to take a second look to see what the email was about, rather than dismiss it as “another email from that marketing site”.

Or did the shorter subject line mean the actual information on the email’s contents was seen by more people?

Or was it a combination of both?

Lesson 2: For future insight, ensure you understand how the subject lines you test actually differ, so you can attribute differences to specific characteristics

The ideal subject line test tells you something you can apply in the future. But many tests look like this:

“30% off new summer fashions” vs “SUMMER SALE 2 Days Only!”

The results reveal a winning subject line for a particular campaign, which is a legitimate and useful outcome. But they give little insight on what it was that made that subject line better.

If subject 2 wins, was it the capitalization, length, urgency, etc. that made the difference?

Without such insights, you have few lessons to apply to future emails.

Net changes disguise negative impacts

And there’s another big danger when more than one thing changes…

Suppose the removal of branding actually hurts results, but the shorter subject line more than compensates for it?

If I then conclude that non-branded subject lines work better, I’m missing out on an important insight: that branding actually helps. Perhaps the best subject is something like this:

(EMR) What YOU see versus what your readers see + how to get email attention

Now let’s make it even more complicated.

Different people may be responding in different ways:

  • The presence of a branded subject line may boost results for some who like my emails: they recognize the email and open it
  • It may hurt short-term results for some who like my emails: they recognize it and file it for later review (better long-term results)
  • It may hurt results for some who are ambivalent about my emails: they recognize it and know there’s no special reason to open it
  • It may have no effect on readers using a smartphone, who might focus much more on the from line anyway
Lesson 3: Try testing on different subscriber segments

Removing branding will see response move in different directions, depending on the recipient. We only see the net effect.

My recent thoughts on reactivation campaigns reflect the fact that subject line elements may have different impacts, depending on the recipient segment.

Novelty versus real effects

If my kids get green pasta, they will show more interest in the meal.

Is it because green pasta is inherently more interesting than any other color pasta? Or is it because it’s different to the yellow pasta they normally get?

Or both?

When you deviate from a regular pattern or approach in your subject line, there are two elements driving response changes.

The first is the impact of change itself: the novelty value.

The second is the fundamental impact of the change: what we might call the true impact.

Both can wear off or change with time, especially the first.

Lesson 4: Be aware of the novelty factor when evaluating results and use it to your advantage

The novelty factor in email marketing means changes like removing the branding or all capitals need reevaluation in a later test to see if the results still hold. Are the results repeatable?

Tests need repeating at intervals anyway – especially if your audience, content, goals etc. have changed and you need to revisit assumptions about what works best.

And, of course, the short-term attention boost that novelty value can sometimes get you is a legitimate tactic in its own right.

Chad White, for example, talks about “wake-up slap” tactics in email design, where deviation from the usual grabs attention. A similar concept might apply to subject lines.

The wider environment

Subject lines don’t work in isolation. They are commonly viewed along with a from line and (often) snippet text, preheader text and/or a preview pane.

So the results of any subject line tests also depend on that context.

I removed the branding from the subject. But…the from line is “Email Marketing Reports” and the preview pane features the site logo and the words ‘”The Email Marketing Reports newsletter”.

So the overall reduction in “branding” and “recognition” is considerably less than if the from line and preview pane didn’t also feature branding.

In the latter scenario, removing the branding might have hurt responses or lifted spam complaints as people had no clue who the email was from.

Lesson 5: Test results may not be transferable between different email situations

OK, let’s stop there as you simply must not write more than 500-700 words in an online article if you want a response (another one of those absolute truths).

Love to hear any of your views on how to better do and interpret subject line tests…comment below or on Google+

Find related articles:

 
Permalink | August 26th, 2011 | 12 Comments »
Get posts like this: RSS feed | via email | via Twitter | via G+

You can follow any comments on this blog post through the RSS 2.0 feed.

12 comments on “5 lessons from a typical subject line test”

  1. Anna says:

    Hello.
    There’s a lot of interesting information in your blog. I am new in IM and I recently started a blog where I, with the help of my mentor Marc Milburn, help everyone who’s interested to set up a successful internet business.
    I can put your knowledge to good use.
    thanks,
    Anna.

  2. I agree Mark – interpretation of results often gets over convoluted and I feel the following are worthy points for consideration in subject lines.

    a) Along with the from name, from address, and pre-header text it forms ONLY part of the recognition elements to your recipients. If you have nurtured engagement in your recipients then recognition will be accompanied by (however slight) a degree of anticipation.

    b) It is a strategic and editorial challenge to create stand out in cluttered inboxes though subject line “crafting”.

    Recognition alone may not promote interaction but most likely will avoid your emails being discarded or deleted.

    c) Sign up to and regularly scan the subject lines in use from your close competitors (and aligned businesses) – you will be amazed at how similar they will be – and it helps you create differentiation.

    d) Do not rely on the results of a single test. Inboxes change by the minute – what worked on one day may flop the next – or even later on that day depending on what else is incoming. And how similar your subject line may be to many others.

    e) Increasingly, emails are being first viewed on mobile devices with smaller viewing areas – so shorter subject lines will be beneficial.

    That’s 5 – but I think there were many more than 5 in your post.

  3. Mark Brownlow says:

    Great points Robin. Especially like C and D.

    I’m often struck when I read the subject lines collated by Chad White at the Retail Email Blog by how similar many are.

    That’s OK where you are not competing with similar senders (I’m just reading consumer survey data which suggests that happens more often that we suspect), but if you are, then identikit subjects hardly help you stand out.

  4. The subject line of an email is what encourages or discourages someone to open. People get lots of email on a daily basis so you need to cut through the noise with an engaging subject line. A/B testing is crucial since every target audience is different.

  5. Adrian says:

    Great stuff Mark, I think a lot of of us sometimes tend to make hasty conclusions, and as you put it, you cant.

    And no need to apologize in your email for testing on us :)

    /Adrian

  6. Mark Brownlow says:

    Adrian: the newsletter headline was going to be “So what does it feel like to be a guinea pig?” but I thought that was a bit cheeky and changed it to asking for forgiveness!!

    P.S. There’s another test running this time, too (but not a subject line one)

  7. Dela Quist says:

    Hi Mark
    Thanks for the mention – some of your readers might be interested in a seminar I presented recently titled Subject Lines: The Hidden Impact on Email Subscriber Behavior for Marketing Profs http://bit.ly/nZSqle in which I expand on the white paper and much more. I think anyone who is interested in improving their SL’s from a strategic rather than tactical perspective should watch it if they can. If you can bear the sound of my voice for an hour it is definitely worth watching. 100% of the 30 people who completed the feedback survey said they would recommend a colleague listen to it and here are examples of why from 2 of the people who watched.

    “Actionable insights that I could try right away”

    “If you think you know subject lines, think again. I had no idea there was so much behind them!”

    It is free to pro members, non members can pay to download the seminar.
    Dela

  8. Chris says:

    Great post, thanks a lot for the detailed info. One thing I must mention however, is that I feel your ‘branding’ is generic.

    Every time I get your newsletter I have to try and remember what it is and who is sending it as it sounds like it could come from anyone.

    If it were something unique, I would recognise what it is instantly. Perhaps this contributed to the higher clicks on the non-branded version?

    Thanks again for all your helpful info.

  9. Mark Brownlow says:

    That’s a great point Chris.

    In fact, it’s not really a branding test at all in the classic sense, not like it would be for “Apple” or similar.

    It’s also a *great* reminder about perceptions. Of course the name is familiar to *me*, but it’s not going to be that familiar to even people who’ve used the site. I’d actually forgotten that! (Note to self: try taking your own advice more often).

    Thanks!

  10. Connor Keating says:

    Thanks for the advice! I find it quite useful and also I’d like to add that I think that following these general guide lines that you have depicted here can have a secondary effect in the form of improving your online reputation. I do not know if this was one of your purposes but this is what I understand.

  11. Not something I had given a huge amount of thought to before. Sure I knew they were important but not to this level! I guess they are no different to blog post titles in that you need to make them appealing to click on in this era of short attention spans!

  12. We’ve found much better results by keeping subject lines as sort as possible and trying to avoid topics that are to ‘sales’ orientated; such as promotion, discount, sale & savings etc. Certainly the U.K market seems to run a mile when faced with a perceived ‘hard sell’