How to A/B test a Shopify store with Google Optimize
4/15/2022 · 10 minute read
Prerequisites:
You know what an A/B test is (if you don’t, take a look at the Intro).
You’ve decided on the change you want to test.
You have Google Analytics installed.
You’re getting significant traffic to your store.
What this guide is:
A straightforward way to use Google Optimize to run an A/B test. With all the best practices for effectiveness, reliability, performance.
What this guide isn’t:
A theoretical or technical approach to statistics. Or a list of ideas to test (since this would require insights from your Analytics).
Intro
Let’s say that you plan on making a change to your Shopify theme.
You’ve seen competitors display a much more brightly colored action button on their hero section. You want to add something similar to your site, because you think it will increase your revenue.
Can you be 100% sure that it will have a positive effect?
Will it make it easier for users to checkout?
Will it decrease the average cart value, by preventing users from browsing and purchasing more products from your store?
One way to find out, is to A/B test this hypothesis. It allows you to draw conclusions about changes you want to perform to your store, with the intent of increasing your KPIs. It’s also called split testing.
The way it’s done, is you try one version with 50% of visitors (randomly picked) and the other with the rest. Then, you try to determine what works best.
How does it work?
I don’t want to bore you with statistics and math, so let’s quickly go over some basic principles.
You assume that the new style of the button will increase your revenue.
This is called a “hypothesis”.
You can’t simply add the button, keep it that way indefinitely, and monitor the results for an arbitrary amount of time. All kinds of factors could affect them, like season or other website changes.
So you test on a “sample”.
In statistics, a sample is a subset of the total (total being all your visitors).
If the sample is too small, the conclusions won’t be safe. You can’t just try adding the button for three users, and draw reliable results.
So you need a large enough sample, so that the results (revenue without button vs. revenue with button) are conclusive.
And when you’ve run the test, you want the results to be “statistically important”. So depending on the difference in revenue per visitor, and the size of the sample, it can be safe to say that, when applying the change (or discarding it), you can be relatively sure that you will observe the same result when you publish the change to all your visitors. This version is called a “winner”.
Some real life examples
I want to give you two real life examples of sites that benefited from A/B testing.
WallMonkeys increased conversion rate by 550%. They tested two consecutive changes on their homepage’s hero section.
Humana increased clickthrough by 433%. They only tested a simpler banner, with a stronger call to action.
A simple change can easily 5x your metrics. Sometimes the original page might be blocking user interaction without you knowing which part is the problem. And that’s what A/B testing helps with.
Some rules
Here are some quick rules, so that your results will be accurate and meaningful.
You can’t run a test if you don’t get enough traffic on your site.
Don’t stop the test before the sample size is large enough.
Only test one change at a time. If you test multiple, you won’t be able to tell which of these changes has lead to the different results.
Don’t always expect that the hypothesis will be proven true. If the test version causes a decrease in the KPI, or no change, keep the original version, record the results and keep testing other factors.
Use common sense when you plan the test, and when you draw conclusions. There might be other factors that you didn’t think of (e.g. the new button slowed down the site). Or problems with your test (the button’s text wasn’t translated properly). Or, metrics that went down when others went up (e.g. clickthrough increased in the test version but average order value decreased). Same for user segments (better results on iOS but worse on Android).
After you set up the test, check the two versions on different browsers and devices.
How to A/B test in Shopify
There are several aspects of a Shopify store you can test. This guide is about changes to the frontend (visuals) of your store.
Other aspects would be product attributes like price and description, shipping, payment methods, emails, and so on.
Why is Google Optimize a good tool for Shopify A/B testing?
Google Optimize is free, while most other tools will cost you hundreds of dollars a month. It’s versatile when used on Shopify, since you can alter your templates in any way you like. It can be used with different devices. There’s tons of documentation for it. It can integrate perfectly with Google Analytics. And, the interface is intuitive and includes lots of features.
If you’re asking, can’t it be topped?
The answer is, maybe. If you need any extra features, another tool might be more fitting. But for the majority of stores starting with A/B testing, Optimize has everything you need.
What’s the best way to test on Shopify, with Google Optimize?
In general, there are two ways you can perform A/B testing.
Server-side and client-side.
Server-side means that determining which version to show, and preparing it, is done on the server, with backend code.
Client-side uses Javascript to perform these after the page is sent to the device, and using the device’s resources.
Server-side is much faster and the delay users experience is insignificant. Client-side will slow down your site a bit.
Unfortunately, on Shopify there’s no real way to perform a server-side A/B test. Mainly because there’s no real way to randomize the picking of versions for each visitor.
This leaves us with client-side testing. And this can delay page load from half a second to several seconds, depending on the implementation.
There are two main ways to implement it:
With the Optimize Page editor (on Google’s dashboard).
With URL redirection: append
?view=to the URL for the test version, having Shopify show a different Template.
#2 is superior.
The difference in performance is significant. Expect #1 to reduce your Performance Score in PageSpeed Insights, by at least 5 more points compared to #2. If you want to know why that’s important, read more about the Shopify speed score.
Also, you have much more control with #2. You customize the test version from within Shopify, instead of Google’s own editor. And, the resulting page is much more stable.
How to A/B test a theme change
This guide assumes you are using the Universal Analytics tag, without the Google Tag Manager. In fact, Google advises against the GTM for this purpose. Also, Enhanced Ecommerce Reporting should be enabled.
Keep in mind that UA will be deprecated within 2023, so you might need to replace the UA code, with the new GA4 code. Hopefully Shopify will support GA4 at that point.
Here are the exact steps:
1. Form a valid hypothesis, based on Analytics and research. For example, I have a button on a hero image that links to a popular collection. The low number of click events on that button suggest that users might be missing it. If I use a more prominent color for the button, more users will browse the collection, and my store will generate at least 20% more revenue.
2. Estimate your required sample size. There are resources and calculators on the Internet to help with this. Keep in mind that the calculation depends on the metric. One calculator doesn’t work for all, nor is the conversion rate the only metric there is.
3. Create your account at Google Optimize.
4. Create a new Experience.
5. Name your experience, add the URL of your Shopify store’s page you want to test (even if it’s one of many product pages), select “Redirect test”, and click “Create”.
6. Click on “Add variant”.
7. Give the variant a name (e.g. “Prominent button color”), paste the page URL of the test page, append ?view=new to it, and click “Done”. In this example the page is Home, but for more advanced cases (e.g. product pages) you’ll need to use the “Advanced redirect”. More on this option here. You’ll also need to change “Page targeting” (e.g. page starts with “…/products/”), in that case.
8. Scroll down and click “Link to Analytics”.
9. Select your property, linked to your existing store, and click “Link”.
10. Under Settings > Optimize, click “View instructions”.
11. Copy the snippet. If you haven’t, install the Chrome Extension.
12. On your Shopify admin dashboard, under Online Store > Themes, select “Actions” and “Edit code” under your theme of choice. It should not be published while you’re making changes.
13. In the code editor, open the theme.liquid file, and right under the <head> tag, paste the below snippet:
{% if template.name == "index" %}
<script src="https://www.googleoptimize.com/optimize.js?id=OPT-XXXXXXX"></script>
{% endif %}The first line describes the template name, to determine if the script should load or not. Since adding the script causes a significant decrease in the load speed, we can’t add it to all templates, if possible.
To replace it, here’s a list of template types. In this case, it’s index because the button we want to test is on the homepage.
On the second line, is the actual script that’s responsible for presenting the test version of your page. Replace it with the snippet you copied from the Optimize dashboard.
Don’t forget to click Save.
14. Still in the code editor, on the sidebar, under Templates, click “Add a new template”.
15. Pick the template you want to test (index for the homepage in this case), name the alternate one’s file with “new”, and click “Create template”. (This guide assumes you’re using Online Store 2.0 JSON sections.)
16. From the sidebar, under Templates, find your original template (index.json in this case) and copy all its contents.
17. Go back to the new template, paste the copied contents (replacing any existing content) and Save.
18. Click on “Customize theme”.
19. On the theme editor, either pick the new template from the dropdown, or add ?previewPath=/the-path?view=new to the URL bar (replace the-path) and hit Enter. The second option is for when it won’t let you pick the new template. In the case of the homepage, you can add ?previewPath=/?view=new
20. This is your test version template. Make any changes according to your hypothesis, and click Save.
21. Back to Google Optimize, under Objectives, click on “Add experiment objective” and “Choose from List”.
22. Click on “Revenue”, or any other objective you want to track. You can add additional objectives too.
23. Time to test the Google Optimize script. Under Optimize Installation, click “Check installation”.
24. The preview theme should open. If not, go back to Shopify, click Preview under Actions, and repeat the check. You should be seeing that “Optimize is correctly installed”. If not, make sure your Google Analytics loads and tracks, the page you opened is the template you added in the if clause, and the snippet was copied correctly. Click on “Back to experience”.
25. You can preview the variants, but you might have issues previewing from Optimize. If you do, you might need to refresh the page, clear cache etc. Instead, you can simply visit the page on Shopify, appending the “?view=new” URL parameter.
26. At this point, you’re ready to get the Experiment running. Click on “Start” above the variants.
27. Click on “Start” again. Your experiment is now running.
28. If your theme isn’t published, it’s time to publish it now.
29. It’s a good idea to confirm the test is working properly. Because the client script uses cookies, if you open the page multiple times from the same browser, it will keep showing the same version. So, you’ll need to open and close the page on an Incognito tab, several times. You can confirm that the test version is displayed by both the content, and the view URL parameter that will automatically added.
30. That’s it! Your test is now running, and you will be notified when a winner is determined.
The report
While your A/B test is running, you can observe its status on the Reporting tab of your Optimize dashboard.
It will present information like what the current metric looks like for each variant, and the overall status of the experiment.
When the experiment is finished, then it’s safe to say whether there’s a winner. At that point, you can look into your analytics, for more insights about your segments.
Conclusion
This was a detailed guide on how to setup and run your A/B test of your Shopify store.
Of course, there’s a lot of research to do before creating your test variant. And it will require a lot of careful studying existing analytics, together will plain intuition. The same is true for examining the results of the test.
If you need help with creating and running your test, or if you have any feedback about the process of this guide, feel free to leave a message.
