As a journalism major in college, I rejoiced in the fact that I tested out of most math classes and would only have to take one math course to complete my degree. Little did I know that a single statistics class would come back to haunt help me in my career as a digital marketer.
Running a successful email marketing A/B test depends on a few elements straight out of your old stats class. But don’t worry, you don’t need to pull an all-nighter to set up your first A/B test, just follow these simple steps:
1. Develop a Hypothesis:
It doesn’t matter if you’re testing a subject line, a call to action, a button, an offer, or something else, you should have a reason for running your test, i.e. something to prove or disprove. A/B testing is more than just throwing two options to the masses to see which performs “better” and assuming it’s the end all, be all. There needs to be logic behind not only why you are running the test, but what the outcome will be. For example:
- A subject line that teases a “new product” will have more opens than a subject line that simply states what the new product is because people are curious about the new product.
- An email body that has 4 or fewer interaction points (links) will have higher click-throughs than an email body that has 5 or more interaction points because too many options creates no action.
- A subject line that uses the brand name will earn more opens than a subject line that does not use a brand name because brand recognition increases opens.
Each hypothesis includes a rationale at the end. Knowing the reasoning behind the outcome, or in other words the reasoning behind subscriber behavior, is just as, if not more important, than which performed better or worse and can lead you to new hypotheses to test.
2. Choose your test group carefully:
To limit variability and maximize the success of your A/B test, you’ll want to perform your test on a similar group of email subscribers. In statistics, this is known as “blocking,” and it helps increase the precision of your tests.
In email marketing, you’ll want to look closely at the list you are going to use. Hopefully, it goes without saying that you’ll want to only use active subscribers, but did you know that removing highly-engaged subscribers may help your test too? Both represent opposite extremes of your subscriber spectrum and could inaccurately sway your results. Other aspects to consider when it comes to your test group include:
- Email service providers: If one or more of the providers you send to regularly causes problems with deliverability, remove them from your test.
- List segments: Unless your hypothesis is dependent on a particular segment, now isn’t the time to slice and dice your list, meaning you don’t want to target just men or just subscribers in a specific zip code.
- Time of send: If you always send your emails at 3 pm on a Tuesday, you should do so for your test too, unless of course time of send is what you are testing.
In short, you’ll want to eliminate as many variables as possible from your email test. Keep things consistent with what you’ve previously done. Rule of thumb is to make one change at a time so you can really zero in on what is causing a change in subscriber behavior.
3. Analyze the statistical significance of your test:
Just because you performed a test doesn’t mean the results are conclusive. A test shows just one data point and often requires additional testing before you can conclusively rely on the results. We often suggest testing a single hypothesis for at least 3 months before determining a winner.
Once you feel you have enough data to determine what we refer to as statistical significance, you can take advantage of online tools, such as this A/B split test calculator. Just plug in the number of subscribers you sent your control and variable to and the number of conversions for each. The calculator then tells you if the results of your test are statistically significant. You could calculate the statistical significance yourself, but because I don’t trust my mathematical skills—again, journalism major.
Remember, when your test is complete and you’ve analyzed the results, you’ve still just got one data point of evidence. Before relying on that data point, you’ll need to do further testing. Depending on your industry and audience, you may need more than 5 statistically significant tests before you can make any assumptions.
Although that may seem like a lot of work, A/B testing can give you the type of insights that can increase open rates, click rates and more, optimizing your email strategy. Happy testing!