I’ve said it many times:
Optimization is a dedicated, repeatable process.
I wanted to take this easy-to-overcomplicate process and break it down to its very foundation.
That’s where this checklist came together.
It outlines everything you are going to need to run a successful optimization campaign — from the necessary technology to the final test analysis.
Let’s get right into it.
Step 1: Necessary Tech
In order to run split tests, you’re going to need some tools.
1. Google Analytics
….or your preferred analytics suite.
This is standard. Without an analytics suite, you have no data and without data you can’t do a darn thing (that’s well informed). Simply put: your analytics data will tell you where you are succeeding and where you are failing.
(RELATED: Applying Website Analytics to Your Digital Marketing)
Use your analytics to identify pages that need work.
This is Google Analytics’ dashboard. Learn the ropes, this will be one of the, if not the most, important tools in your optimization toolkit.
(If you’re a Digital Marketer Lab member, check out the Mastering Google Analytics Execution Plan to start learning everything you need to get started.)
If you don’t have Google Analytics on your site yet, stop reading this right now and go get it on there.
2. Testing Tech
In order to run a split test you are going to need a technology to edit variations, split these variations, and track conversions.
Most of these technologies are about the same, especially at their lower tiered price point. We use VWO, but feel free to use whichever you prefer.
Pro tip – make sure to integrate your tool with Google Analytics to report in GA for better data
3. Qualitative Tools
Qualitative data is incredibly important and severely underused. There are some great tools out there to choose from – start with one and expand out when you run into user knowledge gaps.
TruConversion – A customer behavior analytics tool specializing in passive user data, i.e., user data where the user is unaware they are in a tested environment. We liked this tool so much, we acquired it last year!
TruConversion has heatmaps, session recordings, user surveys, funnel analysis, and form field analysis. (DigitalMarketer Lab Members: Don’t miss out on our DM Deal for TruConversion!)
UsabilityHub – This is a great site that has five different styles of user tests:
- Preference Test
- 5 Second Test
- Click Test
- Question Test
- Navflow Test
Personally, I prefer the 5-second test.
It’s a great way to get a proof of concept for your page content and offering. If people can’t figure our who you are and what they’re supposed to do next in 5 seconds or less, you need to rethink your page!
4. Test Significance Calculator
Here’s a good one from VWO.
All too often I see tests get called that don’t really have significant data. This is mainly due to the major disparity between variants at the beginning of a test. As numbers normalize (and they always do) your win on day one may actually be a loss.
5. Test Duration Calculator
Here’s another good one from VWO.
Yeah…you probably shouldn’t run this test.
You should always know how long a test is going to run.
- If it takes more than 6 weeks it is likely not worth running
- You always need a test timeline
A simple calculator like this takes the guesswork out of your campaigns and helps qualify whether a page is actually testable.
Step 2: Test Research
Every great test needs a solid foundation. Your research is that solid foundation.
Use each step to come up with a test that will provide value regardless of whether it wins, loses, or flatlines.
1. Identify Test Worthy Pages
People ask me all the time:
“Justin, how the HECK do you know which pages to test?”
(That’s not even an exaggeration! After this post went live in December 2015, that’s the question I’ve been hit with the most.)
And for good reason…
Selecting the right page to test is arguably the most important step in the entire optimization process.
The “Test Everything” mantra is not only incorrect, but it’s dangerous.
Even sites with high traffic, and the capability to run a test on every page, should be very careful.
Just because you can test something doesn’t mean you should.
You need to be highly selective about the pages you work on.
During my time optimizing and auditing sites, I’ve found a few pages that are more often than not worth testing.
Pages Not Worth Split Testing 1: Your Cart/Order Form
Why?
If a visitor made it to your cart, then they’ve jumped through all of your hoops. You just need to bring them home!
Unfortunately, the cart is the section of your site where your visitor is the most anxious.
There’s a reason the industry standard for checkout conversion rates is only 2%…our customers need reassurance, and we aren’t meeting these needs.
How to know your cart is your next test page:
Depending on your Google Analytics settings, there are a few ways to figure this out. (We’ll dive into how you should be using Google Analytics to find test pages in just a bit!)
First, look at your eCommerce conversion rate. [Conversions > Ecommerce > Overview]
This might not be as telling as you want it to be. So from here, I’d look at the checkout visits via Behavior > All Pages and use an advanced search for pages that include your checkout URL schema.
From here you can look at the thank you page rates to see if there is a systematic drop off.
Pro tip: If you are seeing great conversion rates on some of your checkout pages but lackluster ones on the other it is likely not a cart problem, but it’s an offer problem.
Pages Not Worth Split Testing 2: Your Paid Landing Pages
Why?
When it comes to your paid media, you have a ton of control. Furthermore, this is the main entry point for your cold audience, so you need to make it count.
How to know your landing page is the next test page:
You need to know a little bit about your expected landing page conversion rate baselines. For example, at DM we expect our Lead Magnet landing pages to convert at 40% at minimum. When we see something drastically lower, then we know this page needs to be fixed/tested.
If you KNOW something is going to increase conversions, you shouldn’t test it – you should implement.
This is exactly what happened when we had a landing page running to a cold audience that was converting at 21%.
This page had several different avatars going to the same landing page. *Slaps head* We knew we had to set up some personalization on the page to change the message on the landing page.
Here’s the standard page everyone saw before we personalized the experience:
And here’s a personalized experience:
Did we test it? No.
Did we see a lift in conversions? Absolutely.
Instead of the standard “10-Minute Social Media Audit” headline and static sub-headline, we reiterated the message in the ad.
In fact, when we added personalization to the page we saw a 58% lift in opt-ins. This was so successful that this has become a part of our ad creation strategy (which I’ll outline more in a future post ;))
Pages Not Worth Split Testing 3: Category/Product Pages
Why?
So none of these pages directly increase sales, but they qualify visitors and move them to the most critical step: the checkout page.
If you are seeing a drop off from your product page to your checkout page then you know your product page isn’t doing its job.
Similarly, if no one is going from your category page to any of your product pages there is something fundamentally broken.
Pages Not Worth Split Testing 4: Thank You Pages
Why?
This is the ultimate cross-sell/upsell opportunity. The thank you page is often overlooked as a selling opportunity. If you aren’t using it to sell, just start. From there you can test the offer or how you position it.
How to know if your thank you page is your next test page:
If you notice that people aren’t taking your thank you page offers, then it’s time to change the aesthetic or the offer itself. If your thank you page isn’t selling anything else, don’t even test it — get an offer on there!
Remember, you need to be strategic with your page selection!
When you are running experiments you are trying to gauge what works and what doesn’t.
An interesting test isn’t a test where you just validate your hypothesis; if you knew there was going to be a lift and there was a lift… well, you only just converted 50% of the traffic you could’ve right out the gate.
An interesting/useful test is when you see results that fall below or exceed your expectations.
Like I said, those 4 pages are ones I’ve seen time and time again not worth testing. But, unless you’ve spent “10,000+ hours in the pool,” you might need a stronger rule of thumb when evaluating the opportunity of your own…
Page Split Testing Guidelines
These are the guidelines you need to follow when you’re determining how worthy a page is to test. Skip testing these pages:
- Your worst performing pages (I know this sounds counter intuitive, but don’t throw your hands up just yet).
- Pages that don’t impact your longer-term business goals, e.g., your 404 page.
- Pages that don’t get enough traffic to run a split test.
So, why shouldn’t you be testing your worst performing pages?
It comes down to the ol’ ‘M’ word…
Math.
You’re job is to focus on opportunity pages.
Pick the page that will have the greatest impact on your goals. If you expect a 10% lift from your efforts, would you rather that lift be on a page converting at 50% or 5%?
I’d choose 50% (that’s a 5% lift over a .5% lift).
However, your sub par pages can be an indicator for optimization needs — it’s important to note that it is ONLY an indicator, not the final word.
Poorly performing pages and test appropriateness is akin to the “All squares are rectangles, but not all rectangles are squares” concept.
There is cross-over, but it comes down to opportunity not poor performance.
Further — your worst performing pages don’t need an “iterative” testing campaign, what they need is an overhaul.
The ship is sinking; you don’t have time to hypothesize over what to do next, you need to make a drastic change that likely doesn’t need to be tested.
Remember, don’t test; implement!
For the same reason you don’t want to test the worst performing pages, you also don’t really need to test your non-conversion oriented pages.
Do you think we run tests on our About Us page? Heck no!
There are some pages that aren’t inherently a conversion page, but can impact some key metrics. The first that comes to mind is the 404 page.
We’ve all seen the classic “dead end” 404 page like the one below:
But optimizing 404 pages has proven to be useful. Put an offer or some additional steps to keep the user engaged. You don’t need to test adding these elements to the page!
Just add content that fits inline with your goals and move on to more important pages.
Formstack does something cool where they don’t even use a 404 – they just redirect you to their homepage!
At DM we try to get people to Lab…
Unfortunately, Harry & David just make it look like the body content of the page is missing, but they still let me navigate the site!
The final page selection element you need to look at is the number of visits and number of conversions your page gets over the potential test period.
This is a qualifying metric and will determine whether you can actually test or not.
Remember if you can’t test, you can still optimize! If the page directly impacts revenue and is a high opportunity page, then you should optimize this page EVEN if you can’t test.
You can identify your pages easily via Google Analytics.
I recommend looking through your funnel in Google Analytics and highlight the drop off areas.
A simple way to do this is to search your intended funnel using the search function above the “Bounce Rate” column. Start with the initial landing page and move down your funnel.
Record the Unique visitors along the way to get an idea about your drop off.
You could also use the Funnel Visualization report, found under Conversion > Goals > Funnel Visualization, or other 3rd party tools if you prefer (like Wicked Reports or Heap Analytics).
As another option, if you want to just view a Traffic Visualization report, you can generate one under Audience > Users Flow…
…where you’ll see something like this:
Ok, back to it.
Next, you’ll want to contextualize the pages. You’ll always see a massive drop off on your home page but that page is so far away from your main converting action that it doesn’t make sense to test.
Now, if you see a massive drop from a product page to the checkout page you know there is something wrong with your product page that you can optimize.
Just jump into Google Analytics and start digging.
When you find a page you think is test worthy, make sure to ask these 4 questions:
- Does the page get enough unique visitors?
- Does the page get enough raw conversions?
- Does this page directly impact my goals? If indirect, how far away from the primary conversion action is the page?
- What’s the potential impact on [insert goal, e.g., sales, leads, etc…]?
These 4 questions will…
1. Qualify the page.
2. Give you an idea whether testing is going to actually be useful.
Phew! That was a mouthful. Now, let’s talk about using qualitative data.
2. Use Qualitative Data e.g….
- Heatmaps
- Session recordings
- User tests, etc.
….to understand user behavior
Analytics are kind of like the tin man: they have no heart. In order to create really stellar variants you should consider the user. Gathering user data is relatively easy and inexpensive.
One of the most basic types of qualitative data involves click tracking, mouse movement, and scrolling. All of this data is reported in what is referred to as a heatmap.
Heatmaps are a visual representation of a user interaction on your site.
You’ll want to get a lot of visitors recorded so you can begin to look for trends. This report will shed light on whether a call-to-action (CTA) is getting clicks or people are consuming your content. This is way more helpful than just looking at your page’s bounce rate in Google Analytics.
Other types of qualitative data include…
- user surveys
- usability studies
- session recordings
- customer service questions
- sales questions
This type of data helps you pick the right elements to optimize on your page.
3. Select the KPI(s) you plan on tracking
If you run a test that only looks at top funnel metrics such as clicks, then you aren’t going to get a full understanding of the actual impact. This is why you need to select your KPIs and know how they translate to your business-oriented goals.
Top funnel metrics are often referred to as micro-conversions and they are very much worth measuring. However, you shouldn’t make business decisions based on micro-conversions.
Make sure to have page level goals as well as campaign level goals for all of your tests.
Obviously I want to see if the immediate change I made on the page generates more leads. Unfortunately leads alone don’t tell the whole story. I wanted to see how each variant impacted sales too.
- Page goal: Leads Generated
- Campaign Goal: Tripwires purchased
This will give you the short view, i.e., what happened on the page, and the long view, i.e., how it impacted your overall campaign.
Pro tip – measure several related metrics, e.g., add-to-cart rate and sales
4. Calculate Your Test Timeline (the test time period)
Every test needs a definitive stopping point.
If you are just testing into perpetuity you are ignoring the possibility that there is no change between variants.
Set your test timeline and stick to it! Use your duration calculator and round up to the next week. For example, if your tech says you would have meaningful results in 10 days, run the test for 14.
Why? Well people act differently on different days and you must account for this variance in behavior.
5. Develop a Hypothesis…
Based on this format:
Because we observed [A] and feedback [B]
We believe that changing [C] for visitors [D] will make [E] happen.
We’ll know this when we see [F] and obtain [G]
A clear hypothesis puts a stop to ad hoc testing. Testing for the sake of testing or for a particular hunch will cause more harm than good.
Following a basic hypothesis format like the one above sets your test’s scope, the segment, and the success criteria. Without a hypothesis you’re guessing and you don’t want to base a campaign’s success or failure on a guess.
Step 2a: Page Spot Checks for Test Inspiration
You might be having a difficult time coming up with a new variant based on your data alone. Here are some elements on a page that might be able to help you with your next variation.
SKIP THIS SECTION IF YOU ALEADY HAVE A VARIATION CREATED
1. Page Content
- You have a strong visual hierarchy, e.g., the eye goes to your most important content/CTAs.
- Copy focuses on benefits and value, not just features.
- There is a strong ad scent, both in page visuals and reiterative copy.
- You don’t overly rely on the subheadline.
- You don’t have a cutesy headline (Must answer who you are, what you do, and why it’s good for me in just 5 seconds).
- Your primary CTA to other link ratio is as close to 1:1 as possible
- If you’re an unknown brand, use trust indicators and prominent logos.
- Logo is in your top left corner
- Use one style text alignment, i.e., don’t start with center alignment and jump to left alignment later.
2. The Call-to-Action Button
- People see (and click) your CTA.
- The button has a contrasting color that makes it stand out.
- There is no competing CTA or information in close proximity to the button.
- There is no generic button copy, e.g., Submit.
- The CTA sets clear expectations (and your content delivers). E.g., If ‘Add to cart’ don’t send to the cart to checkout that’s what ‘Buy Now’ is for.
- Your CTA is above the fold.
- If a long form page you reiterate your CTA periodically
3. Images
- Your image conveys a message and isn’t just a white space filler.
- Avoid stock imagery (or at least use Photoshop to make it less stock photo-esque).
- Stop using images as headlines or for cool text fonts. Text should be text.
- Make sure your image doesn’t slow your load time.
- Avoid image sliders.
4. The Form
- Form field number should be consistent with the perceived value, e.g., if you are selling a 250k water filter you can ask for more than just email.
- Only ask for required information, e.g., if you have fields that are optional you’re likely better off removing them.
- Your form has a headline.
- Your form has a clear CTA.
- Avoid having the field name in the form field, put it above the field.
- Add in privacy policies/trust indicators (pepper these in, don’t put in EVERY trust field possible).
- If you need more than 3 form fields, studies show you can ask for up to 7 pieces of information without drastically hurting conversions.
- Your form tags the visitor and sends them an indoctrination and confirmation email.
Step 3: Pre-Launch
You have your hypothesis, your variations, and your test schedule. You’re almost ready to go! Check off these next few steps then go ahead and hit ‘Start Test’ in your testing tool!
1. Goals Defined in Google Analytics
Just having Google Analytics on your site isn’t enough, you need to set your goals. Setting custom events or ecommerce tracking works too — you just need something to measure.
In this image I wanted to see how one of our individual campaigns influenced sales.
When you have proper ecommerce and/or goals reporting in GA, then you can start to drill down on campaign metrics. This is incredibly powerful and will start to show you the efficacy of your campaigns in a single dashboard.
2. Testing tech custom dimension is reporting correctly.
(For VWO users, this is a great resource.)
Keep in mind you won’t be able to see your test data in Google Analytics, so you’ll be entirely reliant on your testing tech for analysis. If you’re not reporting correctly, you won’t have clean data.
3. Omit mobile traffic (unless a mobile test)
What works on a desktop won’t necessarily work on a tablet or a phone. I always make sure to run my tests on desktop traffic only in order to have the most accurate results.
4. Your Pages Render Correctly in all Browsers
There is nothing worse than broken variants. You think a variation loses because your hypothesis was incorrect, but really it is a tech issue.
I am 100% okay with using WYSIWYG editors if you are making minor changes, e.g., images, copy, buttons, etc…
However, if you are doing major structural changes DO NOT use the WYSIWYG editor – something will break! Here you should just use split URL tests.
Pro tip – Use BrowserStack or preview options in VWO.
5. There is No Page ‘Flicker’
This is another tech issue that mainly occurs if your code is loading slowly or in the wrong spot. The flicker will hurt your results. It would be especially painful if you ran a price test and the customer saw one nice low price then saw another.
That said…don’t run price split tests…
6. There are No Other Test Conflicts…
…e.g., a test on the current page or current tests that get significant traffic from this test page.
You don’t want your tests overlapping. If you run concurrent tests make sure that you omit the other tested traffic from that test to avoid any biases.
7. Both Pages Function as Intended
If one page is showing a broken image or your stylesheet is all wonky, your test will be worth squat.
You might see a lift and your page is broken then the conversion lift (or loss) is not due to the changed variable but the page functionality.
8. All Links Work
Similar to making sure your page is functioning, you have to make sure your links actually work AND go to the right page! A split test between a page with links and a page without links is obviously a broken test!
9. All Non-Essential Links Open in New Tabs
Some people believe that all links should open in new tabs, but if I’m on a product page I don’t want my checkout page to open in a new tab. Use your better judgment here.
10. Variation Load Times are Similar or Identical
Load time is the unsung hero of CRO. If you have a bad load time, your conversions are going to suffer. If you have a variant with a better load time, that variant will likely beat out it’s competition.
Keep your load time in mind when you optimize!
Pro tip – use PageSpeed
Step 4: Calling A Test
In step 1 you defined a test period, the trick here is to stick to that test period. Here’s when you know you can call your test.
1. Your Test Timeline Matches your Schedule
Make sure to actually run your test for the time you scheduled it for.
Do not call a test early because it ‘looks good’! Get the sample size and call the test. Also, don’t just string a test out longer because you didn’t get the results you wanted. When you hit your scheduled end date call it a day.
2. You’ve ‘Completed the Week’
In other words, a test that starts on a Tuesday must end on a future Tuesday.
3. For a Test Win
- Your lift is statistically significant
- Minimum 100 conversions per variation.
4. For a Test Loss
- Your loss is statistically significant
- Minimum 100 conversions per variation.
5. For a Null
- No statistically significant difference
- The numbers have normalized
- The test ran for the entire test schedule
Step 5: Test Analysis
So by now you either have a winning, a losing, or a null test result. Here’s where you dig into the data to analyze what happened during the test period and to come up with your next steps.
1. Report all of Your Findings…
…on a test report sheet or PowerPoint deck.
I break this report into a few sections:
- Slide 1: Test Title, URL, Timeline, Metric(s) measured
- Slide 2: Hypothesis
- Slide 3: Variants Side By Side
- Slide 4: In-depth results
- Slide 5: Results in short showcasing the winning variant image, conversion lift, and confidence rate.
- Slide 6: Analysis
- Slide 7: Other Observations
- Slide 9: Recommendations
2. Report Your CONVERSION RANGE
…e.g. 30% lift ± 3% or expect conversion to be between 27%- 33%.
The conversion range is the range of the lowest potential conversion rate and the highest possible conversion rate. Reporting a static conversion rate is misleading.
When you report a 40% conversion lift and when really you have a range of 35-43% you aren’t properly setting expectations. By the way, if you don’t set proper expectation with your boss/client…you’re going to have trouble.
Don’t let your boss or client think that the conversion rate is static. It isn’t. Set proper expectation by reporting on your conversion rate as a range. VWO’s latest reporting engine does this for you!
3. Look at Each Variant’s Heatmap
This is made much easier if your testing technology has heatmaps built in. This is a great thing to do to fill out the ‘Other Observations’ section of your report and will help you find new things to optimize and test.
4. Analyze Key Segments in Google Analytics
Here, you’re looking to see if the test had more/less impact for certain visitors.
5. Implement the Winning Variation
You know what works! Now it’s time to put it to work. Use your data to make educated decisions about what changes you should be making on the page.
6. If Null, Pick Your Preferred Variation
At this point, if your test has declared no winner from either variation, you can choose which you’d like to implement. Use this data to develop a new hypothesis and create a new test.
7. Use New Learnings
Here is where you can learn from segments, heatmaps, or the test proper to develop your next iteration or fuel a test on a new page.
Optimization is a process. Your latest findings should feed into your future work.
8. Share your findings
This part is really easy to forget. At the very least you should send your report over to your boss/client and your colleagues that had a stake in the test. If you want to go above and beyond you could even publish your findings as your own lead generation device (people LOVE case studies).
If you follow this checklist you will never be caught with your pants down when running a test. This is the checklist I use internally and have used when setting up campaigns for past clients and now it’s yours.
Phew, you’ve got all the info you need to get your split tests and optimization started! Enjoy and Happy Testing!
Have a question?
Ask the DM Team and 7,856 other members in the DM Engage Facebook Group.
Not a DM Lab Member? Learn more here.
