A/B testing, or split testing, allows you to continuously improve your listings and ensure that you have content that converts. In this blog post we will discuss how Amazon’s A/B testing works and the benefits it can provide for Sellers!
Although time-consuming, it is also possible to conduct A/B testing manually or by using third party software options. We have found that Amazon’s testing allows Sellers to obtain helpful and accurate results without dedicating inordinate amounts of time or paying additional fees.
Why Should I Split Test?
Split testing is the easiest and most trustworthy method for businesses to learn what potential customers like and what they don’t. Maybe there’s a small detail in your main image that customers don’t like, or a certain keyword missing from your title. Maybe customers would prefer to see alternate product features highlighted in your A+ Content.
When performed properly, A/B testing can answer all of these questions and take the guesswork out of improving your listing. Testing can increase your engagement and conversion rates by directly leveraging customer engagement.
The level of improvement varies, but we typically see between 1-3% higher conversion rates following successful split testing with a clear winner. On high traffic ASINs, this makes a big difference. If your listing sees 10,000 visitors in a month, every one percent increase in conversions means 100 more products sold. Some of our high traffic listings see 50,000-100,000 visits from potential customers every month. In these cases, just a 1% increase means 500-1,000 more sales.
How to Start Testing with Amazon
Amazon’s built in A/B testing is a free tool for all Brand Registered Sellers. Right now, the feature is only available on high traffic ASINs, and Amazon does not specify what level of traffic is required. The only way to know for sure is to check and see if the ASIN is on the list of eligible products when creating a new experiment. Don’t let that fact discourage you from moving forward. The data gathered from testing eligible ASINs can be extrapolated and used across your catalog.
Follow these steps to get started
- Navigate to the “Manage Your Experiments” section of the Seller Central Brand Dashboard
- On the page all your current experiments and their results. Click to “Create a New Experiment”
- Select either Title, Main Image, and A+ Content from the dropdown
- Choose from the list of eligible ASINs
Although Amazon limits which features are available for testing, we find that the options presented allow you to improve the content with the largest impact on conversion rate. If you are testing an an ASIN for the first time. we recommend starting with the product title. We also recommend testing your top performing listings first, as these will receive the most traffic. This will make the test results easier to interpret because of a larger data sample, and you will see the largest bump in profits after testing.
Crafting every test will be different and entirely dependent on the specific product. Amazon provides you with a few helpful tips for experiments and we feel they are worth reading. Perhaps the most important hints to keep in mind are to make the A and B versions as different as possible and make sure you have a clear hypothesis for each test. If you are testing a title and only change a word or two, then it would be impossible to know if the change was the reason for the results or if it was just coincidence. The same is true of minor image changes and A+ Content tweaks.
Make sure to keep both A and B versions distinct so the test provides the most useable data. Minor Changes mean minor results.
Once you have decided on sufficiently unique A and B versions, you will need to have a clear goal for the test. The data will be most actionable when you treat it like any other scientific experiment. That means having a defined hypothesis that will help you know exactly how to interpret the results. For instance, don’t just say “we think that this new title is good and should get results” or “we think title B is better than title A”. You should be able to clearly outline why you think that a certain title is better, such as “we think that including more keywords focused on ‘product benefit x’ will result in more sales.”
You will need to set a duration for the test, and it can be anywhere from 4 to 10 weeks. We would recommend that you set the duration for as long as you can wait. Longer testing will lead to a larger sample size and more accurate results. Shorter tests are more prone to producing untrustworthy data due to market volatility. Do not declare a winner before the test reaches a 95% confidence interval, even if it appears that one variant is pulling ahead.
Be specific about the actions you’ve already taken to resolve the problem and provide supporting evidence if possible. This evidence could be screenshots of changes you’ve made to the account, Case IDs from support cases you’ve opened, or copies of invoices.
Follow these tips to create successful tests
- Keep both A and B versions distinct so that you know exactly what worked and what didn’t
- Have a clear goal and hypothesis: Don’t just throw two random options on the table
- Set the duration for as long as possible and don’t end the test until Amazon declares a winner
- Make sure you have enough inventory to last the duration of the test
How to interpret Results
Amazon will provide you with weekly updates on your experiment until the end of your chosen duration. Results will likely become more clear over time as the data sample grows.
The probability that one version is better will tell you which version is “winning” the test. If the results indicate there is a 80% probability that Version B is better, that means that 80% of calculated impacts result in increased sales for Version B. We feel that any versions with results over 75% are generally worth using. If the number is closer to 50%, that means that the change will likely have a low impact on sales and may not be worth investing in.
The Projected One-Year Impact section displays the likely results of implementing the change that was tested. It shows the most likely outcome, as well as the best and worst case scenarios. In each of the extremes, the actual impact will be better or worse in only 2.5% of cases.
If you feel that the potential impacts are favorable for your product, then you can simply make the changes to your products in the Manage Inventory section of Seller Central, or through uploading an inventory file.
Frequently asked Questions
Which products should I test?
We recommend starting with high-traffic products that have a decent conversion rate, as these will provide you with the most clear and actionable results.
What results should I expect to see after running an A/B test?
Amazon split tests will vary based on which variant gets better rankings and ultimately more sales. Most of our tests have resulted in useable data that allowed us to increase conversion rates for the test subjects.
When will Amazon pick a winner?
Amazon will not pick a winner until the test confidence interval reaches 95%. This could only take a few weeks, or the entire duration of the test. Do not end the test early even if a winner has been chosen.
My A/B test is over but I don't see a winner?
Not every test will provide actionable results, but an inconclusive test can still provide insight. If you did not observe a clear winner after the test has ended, then the two versions were likely too similar, or the change did not matter to customers and won’t affect customer behavior.
Are there any disadvantages or risks involved with A/B testing?
You may see a small drop in sales while the test is being performed or shortly after the changes have been implemented. This is usually a small price to pay for increased conversion rates in the long run.
Can I test more than one variable at a time?
No. We recommend testing a single variable for a single listing. Otherwise, you will not be able to determine which variable had more weight in the test result. Not to mention that Amazon’s algorithm can become quite finicky when multiple variables are changed simultaneously.
Can I use 3rd party software to perform the test?
Yes there are several companies that offer software used to perform Amazon A/B tests! We chose to use Amazon’s native testing capabilities because it guarantees results accuracy and we don’t encounter trouble manually changing detail page content.
Keep Your Product Momentum Rolling
Amazon competition is growing every day. Sellers that use a set it and forget it approach to their listings will likely not stay on top for long. Split testing allows you to have peace of mind knowing that you are offering the best possible product listing experience for your customers.
Are you ready to start testing your listings? Do you have any questions not covered in this article? Get in touch with our Amazon experts today!
