Email testing II: what to test, traps and surprising results

The previous post explained why testing is so valuable for email marketers…and described ways you can overcome some of the technical and psychological barriers to starting your own testing regime.

Which begs an obvious question: just what parts of an email are best tested? (And don’t anybody say “all of them”).

Once again, email marketing consultant Jordie van Rijn of emailmonday and email marketing strategists Kelly Lorenz and Kristen Gregory of ESP Bronto Software are here to help find answers…

What are the quick testing wins? What should we start with?

Subject line tests are an obvious candidate as they’re very simple to do. Kristen says:

“Testing on subject line is a perfect example of a simple test that doesn’t necessarily involve as much work as, say, a content split on offer type where you’d need two different pieces of creative, possibly approval by certain individuals to do so, set-up for different promotion codes, potential training of customer service, etc.”

Kelly agrees, citing subject line and time of day/day of week testing as typically the quickest and easiest tests to perform.

“I also like to recommend to new email marketers to test creative elements like adding call to action preheaders, call to action buttons and language, as these often provide big, immediate wins.”

Jordie’s also a fan of CTA and button design tests, adding:

“Also test (product) images: those can produce a big lift and are easy to do.”

While subject line tests are probably the easiest, Jordie warns us to be careful with them: winning concepts may not always transfer from campaign to campaign.

“If you are making a plan, look for easy to do tests with a high potential gain. If you combine this with high re-usability, you’ve got the winners!”

“The easiest test is subject line testing, but usually those are not (totally) re-usable unless you are doing a drip campaign.”

The “reusability” of test results is a big issue, of course. It’s no good doing email tests and then drawing the wrong conclusions through a flaw in your analysis or test setup. So are there any common mistakes to watch for?

What common mistakes should you avoid?

There are, of course, technical testing errors that you can make, such as using too small samples or testing two email versions where there’s too many differences between the two to meaningfully interpret any result boosts.

But our experts identified various analysis and management errors that even the lay tester can (and should) easily avoid.

1. Lack of preparation

One common mistake is alibi testing with no real goal or purposeful analysis of the results. Kelly says:

“Don’t test just to test and look good for your boss – doing so wastes your time and provides no benefit to subscribers. Go into every test with a set plan, test multiple times, analyze your performance and implement the key takeaways.”

2. False attribution and generalizations

On the surface, test results seem pretty good pointers to what works better and best. If a green button pulls more conversions than the red button, then you know which button to feature in future email campaigns.

Unfortunately, it’s not always as simple as that.

First, there are many factors affecting email response. So, as Kristen points out:

“…you can’t just assume that this email test performed this way because of the one variable you changed. That’s why you have to continue to test your theories.”

Kelly adds:

“Looking at the test in a silo may cause you to miss the point. As an example, I tend to caution clients that test in the fourth quarter to be cognizant of the seasonality impact.”

Second, it’s too easy to take a test result and assume the result is universally applicable. Or as Jordie puts it, you can’t…

“…take the outcome of one test and declare it holy for all e-mailings.”

He adds:

“That often makes me kind of upset, because you know that that company is making an effort, but doesn’t yet have the insight…what works at Christmas might not work in summer. What works in marketing emails might not in service updates. Re-test and keep it going, it’s a constant process.”

3. Things change

The point about generalization applies to time as well. Subscriber reactions and responses change with time, so what worked before may not work forevermore.

Kristen says:

“…testing is never done! It’s not uncommon to see what works best become stale and need to be replaced with something new and shiny. You can’t get complacent and say that this approach works best even though you last tested it a year (or more) ago.”

4. Not digging deep enough

Perhaps the hardest mistake to avoid is simply not paying close enough attention to the nuances of the results. Analytical skills grow with time and experience and can turn a failed test into a success. Jordie gives us an example:

“…one client didn’t get any positive results by tweaking the current design. Or so it seemed, because after some analysis it was clear that some tweaks had worked for a part of the subscriber base, but other segments cancelled out the effects. Now they are optimizing design for each of those segments.”

Tales from the front line

Testing is usually praised by experts because of the improvements it brings to your success metrics.

One argument against testing is that many such improvements can be made anyway by anyone with a solid understanding of email design, copywriting and subscriber behaviors.

This just isn’t true.

One thing that every marketer eventually comes to learn is that personal experience and intuition is not a flawless predictor of how subscribers behave.

To illustrate the point, I asked our experts to give us some examples of surprising or impressive test results. So to finish this mini-series, here are their stories:

Jordie:

“We were doing a continuous subject line test, testing four subject lines against a control each month. The control [Newsletter company name month] won or was equal every time.”

“So after 5 months a team member said “Let’s just face it, people want the boring subject line. We can’t improve results like this.” The next month one of the subject lines outperformed the control by +30% clicks. It just goes to show that you should stick with testing.”

Kelly has three examples for us:

1. “One client that’s a large clothing retailer had been sending one big image emails when we decided to test splitting the email into a hero image and a corresponding copy section that was plain text to combat ISPs images-off default.”

“It turned out after testing multiple times that the big image email won across the board! Our best estimate on why was that their subscribers wanted the main point only; they wanted to click through right away based on seeing the clothing and headline.”

“We kept a preheader call to action to help combat deliverability and image-suppression issues and continue to test adding other elements to the emails (like social and mobile), but the results of this one were certainly surprising.”

2. “With an educational goods client, we’ve tested a number of creative elements and subject lines focused on boosting response rates from opens through conversions and in a number of the tests the results were completely deadlocked – no winner to be found!”

“What that said to me was that their audience was responding or not based on variables beyond the content of the emails, which caused us to take a step back and look at the email program as a whole.”

3. “I love doing tests on behavior-based triggers like abandoned cart messaging to determine the best time and incentive to achieve the highest lift. ”

“One client I work with tested sending 24 hours post-abandonment against 3 days and actually saw higher conversions and revenue sending 3 days later, which is contrary to best practices.”

Kristen:

“One of my favorite testing stories involves a gourmet food company that I work with regularly. They were only sending their emails on specific weekdays. I think this is typical for a lot of companies – only sending on Tuesdays and Thursdays, for example.”

“This gourmet food company decided to try an A/B split on send time, pitting their normal weekday send against a Sunday send. They were extremely surprised to find that Sunday’s send totally crushed the weekday send!”

“Ultimately, this result isn’t too surprising as people may have more time to browse and shop on a Sunday. To top it off, getting more orders on Sunday enabled the company to get a head-start on packing and shipping these orders first thing Monday morning. I highly recommend marketers try out some weekend split tests to see if they get better results as well!”

“Another interesting testing story relates to a beauty product client of mine. Every week they have a sale at a specific time and were sending out a generic email reminder to check out the sale on the site. I recommended that they actually try highlighting the sale item in the email and subject line to see if they got more interest that way.”

“They have been running a series of tests and, so far, by simply looking at the opens and clicks, you would determine that the generic approach has been more successful.”

“A closer look, though, shows that the messages featuring the products have brought in more revenue!”

“Think about it: the generic email requires subscribers to open and click in order to find out if they are interested in the product in the first place, while recipients of the product-specific message don’t need to open if they don’t like the item in the subject line.”

“You could also hypothesize that the specific email catches the attention of those readers who might not bother to click to see this week’s product in the generic version. This client plans on continuing to test these two versions and gather more data before deciding which path is best.”

If you’ve any testing surprises or insights for us, let us know in the comments!

Find related articles:

• Channel: Email marketing testing

You can follow any comments on this blog post through the RSS 2.0 feed.