[RANT] 7 Reasons I’m Done Talking About Split Testing

Let me serve you up some ‘Member Berries real quick and remind you about the testing scene back in 2010.

This was the beginning of the testing blitzkrieg and the conversion rate optimization (CRO) movement. The topic of A/B testing was the next “Big Thing,” and everyone was interested in case studies and the best ways to test their site.
Unfortunately, this led to very hyperbolic test results being shared without the proper procedure, which created some VERY unattainable expectations.
Fast forward to today and people aren’t talking about split testing results anymore, they’re talking about CRO processes. Why?
Simply put: Everything that can be said about split testing has been said.
I’ve been talking about and training organizations how to split test for the last seven years. Even though it is a necessary element in a marketer’s tool belt, I refuse to talk or write about testing again.
And here’s seven reasons (one “Why” for every year I’ve talked about it) why.

1. It’s Been Done… A Lot.

Here’s what Google shows me when I search for “Split Test Guide.”

These are the first 10 results, but it doesn’t stop there. You will see some kind of “Ultimate Guide” to testing through page 20 (and likely further, this is where I stopped).
I’ve read most of these guides, and to be honest, they’re all really the same.
They’ll cover a pretest strategy, the minimum requirements for testing, and how to know if results are valid. Depending on the author, you’ll get more focus on one part of the testing strategy, e.g., if you’re reading from an agency or testing technology you will see an increase focus on testing velocity.
In short, you can pick any article and get the basic scope for the testing process.
Shameless plug: My checklist is a pretty good starting point.
The split testing topic is saturated, and there isn’t much of a point for writing a new strategic post, which brings me to my next point!

2. Scientific Method + Math = It’s Not Changing

It’s been said that there are two constants in life: death and taxes.
Well, I’m going to add two more: the scientific method and proven mathematics.
Since these are constants, they aren’t going to change! If you understand your testing method and understand the math (let’s only focus on the applied statistics, you don’t need to be a statistician), then you know everything you need to know about split testing!
I’m gearing up to update my Optimization and Testing certification, and as I’ve been going through the outline there are massive portions that are evergreen (strategies that don’t age quickly); namely the process and testing stats.
So, to save you some time digging through the seemingly unending amount of information surrounding split testing, let me give you everything you need to know RIGHT HERE.

Process

1. Identify your KPIs.
2. Dig into analytics data to find underperforming pages.
3. Do user research to figure out why the page is underperforming.
4. Prioritize which test you should run first.
5. Develop a hypothesis for the test case.
6. Schedule the test case: If it can’t be done within 4-6 weeks, DON’T TEST.
7. Don’t shorten or lengthen the test period.
8. Look at results to see if significant: Testing techs will do this, but you can calculate yourself).
9. Identify if the results are scalable: Is this an accurate subset of all of your traffic?
10. Launch winning variant to appropriate segment — can be all traffic.
11. If satisfied with lift, move on to other pages. If you’re unsatisfied, dig into the results to find more ways to improve the page.
12. Repeat.

Math

1. The “Percent Lift” that is referenced in case studies is the relative change between variants. A 30% lift isn’t an increase from 1% to 31%, it’s an increase of 1% to 1.3%.
2. Confidence Interval: This is the amount of error allowed in testing; you want this interval to be as small as possible. To have a small interval you need more tested cases.

3. Confidence Rate: This just tells you whether there is a recognized difference between the two versions. It’s answering the question, “Assuming there is no underlying difference between A and B, how often will we see a difference like we do in the data, just by chance?”
4. Time Period: Test for full weeks and for a period that is in line with your buyer cycle (very important for B2B).
5. Sample Sizes: Aim for, at minimum, 100 conversions per variation. That’s MINIMUM, for higher risk tests I recommend 200-250.

If you want more math information, you can read it here. However, I’ve seen too many optimizers go down the rabbit hole when it comes to getting into the stats (and in most cases these people aren’t statisticians).
Remember, to always monitor the conversion rate of the winning variant post-test to ensure that the reported lift range is accurate. If you see a dip, you may have seen a false lift and have to get back to work!
Pro tip: Big lifts early on will skew a test, let it run for the time period you allotted to get more accurate results. Big lifts require smaller samples sizes, and in turn require less time. However, you’ll notice that those big lifts vanish as time goes on.

3. It’s a Validation Tool, Not Optimization

If I’ve said it once, I’ve said it a thousand times…
Split testing is not the same as optimization!
Split testing is a tool you use while in the act of optimizing! This is a way to test out your new ideas and see how effective (or ineffective) they actually are.
I don’t want to waste time writing more split testing posts because there are some really cool innovations happening in the optimization space that deserve way more attention and will actually improve your bottom line faster!
Testing also delays your ability to exploit the knowledge you’ve gathered.
If you have to wait three weeks to validate something rather than rolling out you’re likely missing out. You should identify the risk of not testing and if the rewards outweigh the risk roll with it!
Here’s an example:
At DigitalMarketer, we wanted to find a way to increase the amount of time on a page after someone from a Facebook ad signed up for one of our Lead Magnets. We’ve learned from previous split tests that when we add a video to a page, people stay much longer.
So we launched this variant:

We saw page depth increase by 2 and time on site increase by 2 minutes. This helped improve our relevance score and also decreased our ad cost. Based on what Facebook was looking for, we were able to develop a more engaged audience post and ad click — in turn we were rewarded with cheaper ads that were considered more relevant.

4. Finite Number Of Tests

There are only so many tests you can run in a year.
For the 1% of sites out there that can isolate their traffic from test cases while running concurrent tests, they definitely have the capability to run more tests. However, if we stick to the “Always complete the week” rule, then there are only 52 weeks to run potential tests.
Sure, high traffic sites can run concurrent tests at any step of the funnel, but for the rest of us that isn’t possible.
In so many words, I’m trying to say that test velocity is absolute crap. We only have a finite amount of time, a limited number of pages that actually matter, and a finite number of resources to launch meaningful tests.
Let’s look at the test process real quick to see how long a test ramp up period can be (Note: This is conservative).

In this case it would take nine days for me to launch and QA a new split test. This process can be shortened if you have a small scale change, but let’s be honest, small scale changes provide small scale results. Yes, those “I increased conversions by 400% by just changing my CTA” case studies are utter garbage.
Assuming you have enough traffic to run a conclusive test in two weeks, but don’t have enough traffic to isolate tested audiences, i.e., run concurrent tests, then you have a total of 26 total tests you can run a year on your website.
Note: Landing page tests don’t count since they are more of an “offer island” and won’t impact most website tests (unless you are testing in the bottom of the funnel).
Again, you get 26 total website tests. Choose wisely.

5. Tests Can Be Short Sighted 

Test results are only a glimpse into a specific point in time.
Things change! The type of traffic hitting your page changes, the preferred design schemas change, your offering could be tired and less attractive, etc.…
Overall, test results are degenerative over time (just like anything on your site). You can’t just set a winner and forget about it because it was conclusive in March. You need to monitor, retest, and in some cases, blow up entire pages to stay relevant digitally.
A lot of testing processes promote an iterative approach. While there is nothing wrong with this approach in theory, I have to stress that you must iterate responsibly!
Small scale changes and iterating into perpetuity will lead to a lift ceiling referred to as the “Local Maxima.” We hit that with our long form sales letters and couldn’t get a conclusive bump while trying to iterate.
When did we see the bump? When we blew that bad boy up.

Finally, you shouldn’t focus on individual results to make your decisions. I run tests in sequences to show that my changes are “offer agnostic.” A single conclusive test won’t change your business. A series of well-organized tests will.

6. There are Traffic Limitations

Not everyone can or should split test!
One of the main reasons people shouldn’t test is due to traffic limitations. If you run tests on small sample sizes, well, you aren’t going to have accurate data.
If you work off of inaccurate data, then you’re going to have a bad time… and make bad business decisions.
I’ve seen hundreds of cases where someone shares a split test case study and they are surprised when the results don’t hold up after the test has finished. When I see the samples size, I’m no longer surprised!
Would you make a business decision based on a difference of three to four conversions? Then why the heck would you set up a test where you need to do this?!
Here’s a fancy chart, if you can’t get the number of conversions on that page per day, either increase the test period or move on!

7. There Are Better Things To Focus On!

This might be the biggest reason why I won’t write or speak about split testing anymore!
There is just so much cool stuff out there to talk about; I’d be doing YOU a disservice discussing the same ol’ testing strategies you’ve heard about 10 million times.
I actually ran a series of tests where I realized that best practices are actually a useful optimization tool, and you can increase conversions by implementing them on pages where they were previously missing.
This was a huge win for me!
Instead of having to test minutia, I could try new technologies and test their effectiveness rather than wasting time proving if adding “Free” to my CTA would move the needle.
Similarly, if I stop chatting about split testing so much then I’ll be able to share new tactics and technologies you can use to further optimize your site! Cool right?
Here are some things I’ve found out recently that didn’t require testing, but significantly moved the needle:

1. Adding a “Share this [insert offer] and get [insert reward]” to Thank You pages dramatically increases traffic, leads, and sales (last year we saw an extra 70k in Traffic and Conversion Summit ticket sales by doing this).
2. Exit pops don’t work as effectively as scroll pops on content.
3. Any sort of pops on your site must be relevant to the current content on the page.
4. Qualitative data is one of the most powerful ways to improve your site.
5. Session recordings, while time consuming, are a valuable set of data due to the passive nature of the participant, i.e., they don’t know they’re in a testing environment.
6. Videos will increase engagement metrics on pages and this can help reduce ad costs.

I have plenty more, and I should probably turn these all into blog posts…
Okay, let’s end this!
You still need to split test!
Just because I’m not talking about it anymore doesn’t mean it doesn’t carry any weight. However, you need to think of testing as arithmetic and optimization as calculus. In order to properly optimize you need to know testing, much like in order to do calculus you need to know arithmetic.
The math isn’t changing and the process isn’t changing. Once you master it, you’ve mastered it, and you can move to learning more useful optimization strategies.
As for case studies and sharing the tests I am currently running here at DM, I’ll still definitely share those!
That said, testing should become something you do, not something you talk about. Think of it like exercising: When you first start doing it… you want everyone to know (looking at you CrossFit). The minute it becomes a part of your routine, you stop talking about it and start showing results.
So do your research, test your ideas (where possible), and implement the winners! Stop wasting your time reading the 100^th split test procedure blog post and get to work :).