Email testing I: the benefits and how to overcome testing barriers

Testing is like exercise.

Everyone agrees it’s a jolly good thing to do.

Most people never get round to it.

But those that do are immensely grateful they did.

So how do you overcome the barriers and inhibitions to testing? What should you test? What traps should you avoid? And what benefits can you expect to get?

All questions I’m uniquely unqualified to answer. So I asked some specialists to help tackle the issues.

In Part 1, we’ll look at what’s stopping you testing, how to overcome those constraints (even if you’re short on resources and technology), and the benefits testing brings.

In Part 2, we’ll explore what exactly you should test, highlight mistakes to avoid, and reveal some surprising test results.

Our guides are Jordie van Rijn, Kelly Lorenz and Kristen Gregory.

Jordie has over 10 years of hands-on experience as a loyalty and email marketing consultant. He’s also co-founder of Emailtestbox and author of the European email vendor guide.

Kelly and Kristen are both top email marketing strategists at ESP Bronto Software, working with major corporations to optimize their campaigns. Kelly recently spoke about email testing on eMail Radio.

So why don’t we test? Do you recognize the excuses?

There are three common reasons we don’t test:

• no time
• lack of skills/tools/resources
• limited understanding of the benefits of testing

For many of us, there’s a thin line between a reason and an excuse. But the overriding problem is often simply the work that testing demands. Because there is a real time/resource cost involved.

As Kristen notes:

“…a lot of marketers don’t test because it can take extra effort, especially if the technology they use to send email is limited or if processes are cumbersome. Marketers are often wearing so many hats, testing gets put on the back burner.”

Kelly adds:

“…effective testing is not easy and can be time-consuming. In addition to time and resource constraints, I think many times marketers don’t test at all because they’re not sure where to start: it can often be ignorance on the importance of testing and what to test.”

Jordie says many marketers view testing as a later step in the evolution of their email marketing efforts…something that comes after segmentation, for example. But:

“…you can start testing the day you start sending your first newsletter.”

Testing is ultimately about improving the quality and results of your campaigns. So it has an important role to play in a digital environment where standing still means going backwards. Kelly tells us:

“You can’t improve by doing the same thing in perpetuity. In fact, I’ve seen firsthand that staying with the status quo actually hurts in the long run. Results decline, subscribers stop responding and revenue is depressed.”

Jordie agrees:

“The thing is, average performance with your email marketing will get less and less results. The public is (directly and indirectly) demanding you be awesome in the things you do.”

…and testing is one way to move from average to awesome.

So how do you overcome the constraints standing in the way of your testing efforts?

Breaking down the testing barriers

The best way to encourage or justify testing is to understand the benefits it brings.

I’ve published lists of inspirational test results for emails and calls to action, and here are some examples:

• A subject line test doubled clickthrough rates
• A sign-up test lifted opt-ins by 1300%
• A call-to-action test led to 67% more clicks
• An email design test lifted conversions by 200%
• An email copywriting test raised clicks by 51%

As Jordie confirms:

“You can get staggering effects. Some clients have doubled their clickthrough rates by testing CTAs or added €30,000 in revenue by testing different offers.”

A side-benefit also comes from the mental flexibility that testing encourages. Kelly explains:

“I think the biggest lift comes from removing all preconceived notions and being open to testing everything. Clients that consistently test and push the envelope have seen consistent incremental growth in performance.”

Of course, not every test produces spectacular gains. Jordie warns:

“There are no guarantees in testing…You should be able to say, ‘I think this and this version can outperform the current version’, but your test versions might just perform somewhat the same.”

…and Kelly adds that, inevitably:

“…some tests don’t work, but trying different ideas always pushes the needle.”

To get more out of testing, Kelly also recommends getting a general education in email marketing innovations and best practices:

“Marketers should be reading blogs and articles on email marketing to help inspire and get the ball rolling. Taking just 10 minutes a day to read a post or two should help get marketers on the right track.”

While the prime reason for testing is the associated performance improvements, Kristen suggests it can also be…

“…a great way to try something new that others may not be fully comfortable with or to settle an issue that divides coworkers. Don’t argue over who is right and who is wrong or which is better – let the subscribers (and the corresponding data) tell you!”

An understanding of the potential benefits typically smooths the path to finding time and/or resources to actually conduct tests. Then it’s a question of commitment and planning. As Jordie says:

“Make a plan, that is all that it takes…get commitment from management for testing for an extended period of time and just do it! If you run into a problem, there is always an answer, but commitment is mandatory.”

Kristen adds:

“Build time into your schedule where you can brainstorm subject lines or offers – even consider creating friendly competitions in the workplace for the best subject lines/offers/time of day for your business.”

But what about technical barriers? What if you have a small list or no access to the kinds of automated testing tools offered by higher-end email marketing software and services?

Coping with technical barriers and small lists

First off, you may be surprised at how many email marketing services now have easy-to-use tools that support testing.

Even traditionally value-priced services commonly offer the ability to split your list into two random groups and send a different version of your email to each to see which performs best.

If that’s the case, then it’s just a question of using the available tools to implement your new-found testing enthusiasm.

If not, says Kristen:

“The only thing you really “need” to test is the ability to arbitrarily split up your list. If your email platform isn’t advanced enough to do some simple split-sends for you, then go ahead and send out two separate sends manually.”

Kelly adds:

“It’s more work and you may not be able to test as often, but it also draws focus on prioritizing goals.”

Kristen reinforces the last point, advising businesses with resource constraints to:

“…tackle less complicated tests, such as subject line splits, changing only the wording of your main call-to-action in your email, and so on.”

Of course, to do so easily means you need to be able to modify your emails or templates yourself. Kristen’s recommendation for small organizations is NOT to rely on a third party to create emails that are image-based and/or require dedicated help to make tweaks:

“Instead, send more html-text-based emails and, ideally, have easy-to-use templates created so that anyone can make changes on the fly without wasting a lot of time.”

A bigger concern for many is list size. The alternative versions of an email need to be sent to enough people to make the results statistically valid. Otherwise you end up making decisions based on flawed data.

So those with small lists wonder if they have enough subscribers to make testing meaningful, even if they use a big chunk of the list as a test group.

We’ll ignore the definition of what list size is “too small” for the moment and look at ways to compensate for this problem and ensure you get meaningful test results.

Three suggested approaches are:

1. Flip testing

Jordie suggests splitting the list into two, doing the test, then reversing the groups and repeating the test on the next send:

“That doubles your test group size.”

Of course, the email element you’re testing needs to be one that carries over sensibly from one email to the next, like a button color test.

2. Deployment testing

If you can’t split a list at all, then Kelly says an imperfect alternative is to vary your email from deployment to deployment rather than within a single send. So the weekly email might feature a red button in Week 1 and a blue button in Week 2.

The problem, she says, is that results are then affected by factors other than the tested element, but:

“…testing multiple times will help reduce this concern.”

3. Retrospective testing

Insights can often fall out of a simple review of past email campaigns to pick up on any elements commonly associated with particularly good or bad results.

Do you always get an open and conversion rate boost when you mention “free shipping” first in the subject line? Do long articles pull more clicks than short ones?

You can see an example of such a review exercise here.

In Part II, we’ll get our experts’ opinion on what tests get you a quick win and what common testing mistakes you should avoid. And they reveal some surprising test results to whet your appetite for your own testing efforts.

Find related articles:

• Channel: Email marketing testing

You can follow any comments on this blog post through the RSS 2.0 feed.