A/B testing, also referred to as split testing, is a marketing strategy that entails comparing two versions of a webpage or application to ascertain which one performs better. The two versions, A and B, are presented randomly to users, with some being directed to version A and the others to version B. An analysis of the outcomes using predefined indicators such as conversion rate determines which version, A or B, outperformed the other. Essentially, A/B testing allows you to determine which version garners the most clicks, subscriptions, purchases, among other metrics, which can then be used to optimize your website for improved conversions.
For which types of websites is A/B testing applicable?
A/B testing can benefit any website that has a measurable goal, regardless of whether it’s an online store, news site, or lead generation site. When it comes to lead generation, A/B testing can make use of information like the age range or gender of prospective clients in emails aimed at boosting sales. In a media context, editorial A/B testing is used to test the success of different content categories, such as headlines, with the target audience. In contrast to sales-oriented A/B testing, this has an editorial function. In e-commerce, A/B testing is used to measure how well a website or online commercial app sells its products. This involves looking at the home page, product page design, and all visual elements that contribute to completing a purchase, including buttons and calls-to-action.
Which A/B Test is good for you to use ?
There are several types of A/B tests. You should choose the one that best fits your particular situation
Classic A/B test
A classic A/B test is a type of A/B testing in which users are presented with two variations of a webpage at the same URL, allowing for a direct comparison of the two versions. In a classic A/B test, users are randomly assigned to one of the two versions, and their behavior is measured in terms of predefined objectives, such as clicks, purchases, or subscriptions. This type of testing can be useful for comparing different versions of the same element, such as a headline or a button, and determining which version performs better.
Split tests or redirect tests
Split tests, also known as redirect tests, are a type of A/B testing that involves redirecting traffic to one or more distinct URLs. In a split test, users are randomly assigned to one of the URLs, which may contain different versions of a webpage or even completely different webpages. This approach is particularly useful when testing new pages hosted on a server, as it allows for a direct comparison of different versions. Split testing can also be used to compare different landing pages or marketing campaigns and measure their effectiveness in terms of predefined objectives.
Multivariate or MVT test
Multivariate testing, also known as MVT (Multivariate Testing), is a type of A/B testing that measures the impact of multiple changes on the same web page. In a multivariate test, several variations of different elements on a webpage are tested simultaneously, allowing for a more comprehensive analysis of user behavior. For example, a multivariate test might involve testing different versions of a banner, the color of text, and the presentation of a page, all at the same time. This type of testing can be particularly useful for optimizing complex webpages or pages with multiple elements, and can provide insights into which combinations of changes produce the best results. However, multivariate testing can be more complex to set up and analyze than simpler A/B tests.
Regarding technology, there are different options available:
Use A/B testing on websites
You can utilize A/B testing for websites, which allows you to compare version A and B of a webpage, and subsequently, analyze the results based on predefined objectives such as clicks, purchases, subscriptions, and more
Use A/B testing for native mobile iPhone or Android applications
A/B testing is more complex with applications. This is because it is not possible to present two different versions once the application has been downloaded and deployed on a smartphone. Workarounds exist so that you can instantly update your application. You can easily modify your design and directly analyze the impact of this change.
Use server-side A/B testing via APIs
Server-side A/B testing can be performed using APIs, which are programming interfaces that allow for the exchange of data between different applications. APIs enable the automatic creation of campaigns or variations from stored data.
A/B testing and conversion optimization
A/B testing and conversion optimization are two techniques that businesses can use to boost their profits by generating more revenue from the same amount of traffic. Given the high costs of customer acquisition and the complexity of traffic sources, it makes sense to first focus on maximizing the potential of your existing traffic.
The conversion rates for e-commerce sites remain low, with an average of 1% to 3%, which can be attributed to the complexity of the conversion process. Several factors influence conversion rates, including traffic quality, user experience, offer quality, website reputation, and competition. E-commerce professionals strive to minimize any negative impact these factors have on the buyer journey. A/B testing is one method that can help establish a conversion optimization strategy by using data to make informed decisions. However, A/B testing alone is insufficient for understanding user behavior, which is critical to understanding conversion problems. To gain a comprehensive understanding of users, it is essential to supplement A/B testing with other methods that provide additional information. This approach can help you generate hypotheses to test and develop a better understanding of your users.
Numerous sources of information are available to help you obtain a more comprehensive understanding:
Web analytics Data
Data from web analytics can be useful in highlighting conversion issues such as shopping cart abandonment, and can assist in prioritizing which pages to test first, although it does not necessarily provide insights into user behavior.
Heatmap and session recording
Heatmaps and session recordings provide insight into how users interact with various elements on a page or between pages.
Client feedback.
By collecting feedback from clients, such as opinions listed on the site or questions submitted to customer service, companies can gain valuable insights. This feedback can be further analyzed through customer satisfaction surveys or live chats
What are some ways to come up with ideas for A/B tests?
In order to identify conversion problems and gain an understanding of user behavior, it’s essential to complement your A/B tests with additional information. This analysis phase is critical and can be supported by the disciplines mentioned above. A correctly formulated hypothesis is the first step towards a successful A/B testing program and must follow these rules:
Be linked to a clearly identified problem with identifiable causes
Mention a possible solution to the problem
Indicate the expected result, directly linked to the KPI being measured
For example, if the problem is a high abandonment rate for a lengthy registration form, a hypothesis could be: “By shortening the form by deleting optional fields, the number of contacts collected will increase.”
Which elements of your website should you subject to A/B testing?
The question of what to test on your website is frequently asked as companies struggle to understand their conversion rates, whether positive or negative. If a company is confident that their users understand their product, they may not prioritize testing the placement or color of an add-to-cart button, as it may not be relevant to their conversion issues. Instead, they may opt to test the phrasing of their customer benefits. As each situation is unique, rather than offering a comprehensive list of elements to test, we provide an A/B testing framework to help identify these elements. Here are some recommended starting points
Titles And Header
Business Model
Princing
Algorithm
Landing Pages
Page Structure
Images
Navigation
Forms
Call to action
Understanding A/B testing statistics
The test analysis phase is critical, and the A/B testing solution must provide a reporting interface that includes conversion data for each variation, conversion rate, improvement percentage, and statistical reliability index. Advanced solutions offer segmented results based on various dimensions such as traffic source, location, and customer type.
Before analyzing test results, it is necessary to achieve a sufficient level of statistical confidence, typically 95%, indicating that the probability of chance differences between variations is low. This threshold varies based on factors such as site traffic, initial conversion rate, and modifications made, and it can take a few days to several weeks to reach. For low-traffic sites, testing with higher traffic is recommended. Drawing conclusions before reaching the threshold is pointless.
Statistical tests rely on a sample size approaching infinity; hence, exercise caution while analyzing results if the sample size is low, even if the test indicates reliability above 95%. A low sample size can lead to results being significantly modified by continuing the test for a few more days. Therefore, it is advisable to have a sample size of at least 5,000 visitors and 75 conversions per variation. Scientific methods can calculate the sample size, or you can use an A/B testing calculator.
There are two types of statistical tests available for A/B testing:
Frequentist tests, such as the chi-square method, are objective and allow for analysis of results only at the end of the test. The reliability of this method is 95%, and the study is based on observation.
Bayesian tests are deductive and use the laws of probability to analyze results before the end of the test. It’s important to correctly read the confidence interval to make informed decisions. Bayesian statistics offer several advantages for A/B testing, and you can learn more about them in our dedicated article.
It’s recommended to leave the test active for several days to account for differences in behavior observed by the day of the week or even the time of day, despite having sufficient site traffic to obtain a large enough sample size. A minimum duration of one week is preferable, and ideally, two weeks. The test duration can be longer for products or services that require a longer buying cycle, especially for complex products or B2B. Therefore, there is no standard duration for an A/B test.
Tips and best practices for A/B testing
Below are several best practices that can help you avoid running into trouble. They are the result of the experiences, both good and bad, of our clients during their testing activity.
Ensure data reliability for the A/B testing solution
Conduct at least one A/A test to ensure a random assignment of traffic to different versions. This is also an opportunity to compare the A/B testing solution indicators and those of your web analytics platform. This is done in order to verify that figures are in the ballpark, not to make them correspond exactly.
Conduct an acceptance test before starting
Do some results seem counter-intuitive? Was the test set up correctly and were the objectives correctly defined? In many cases, time dedicated to acceptance testing saves precious time which would be spent interpreting false results.
Test one variable at a time
This makes it possible to precisely isolate the impact of the variable. If the location of an action button and its label are modified simultaneously, it is impossible to identify which change produced the observed impact.
Conduct one test at a time
For the same reasons cited above, it is advisable to conduct only one test at a time. The results will be difficult to interpret if two tests are conducted in parallel, especially if they’re on the same page.
Adapt number of variations to volume
If there is a high number of variations for little traffic, the test will last a very long time before giving any interesting results. The lower the traffic allocated to the test, the less there should be different versions.
Wait to have a statistical reliability before acting
So long as the test has not attained a statistical reliability of at least 95%, it is not advisable to make any decisions. The probability that differences in results observed are due to chance and not to the modifications made is very high otherwise.
Let tests run long enough
Even if a test rapidly displays statistical reliability, it is necessary to take into account the size of the sample and differences in behavior linked to the day of the week. It is advisable to let a test run for at least a week—two ideally—and to have recorded at least 5,000 visitors and 75 conversions per version.
Know when to end a test
If a test takes too long to reach a reliability rate of 95%, it is likely that the element tested does not have any impact on the measured indicator. In this case, it is pointless to continue the test, since this would unnecessarily monopolize a part of the traffic that could be used for another test.
Measure multiple indicators
It is recommended to measure multiple objectives during the test. One primary objective to help you decide on versions and secondary objectives to enrich the analysis of results. These indicators can include click rate, cart addition rate, conversion rate, average cart, and others.
Take note of marketing actions during a test
External variables can falsify the results of a test. Oftentimes, traffic acquisition campaigns attract a population of users with unusual behavior. It is preferable to limit collateral effects by detecting these kinds of tests or campaigns.
Segment tests
In some cases, conducting a test on all of a site’s users is nonsensical. If a test aims to measure the impact of different formulations of customer advantages on a site’s registration rate, submitting the current database of registered users is ineffective. The test should instead target new visitors.
- Conduct an acceptance test before starting to ensure that the test is set up correctly and objectives are defined properly.
- Test one variable at a time to isolate its impact accurately.
- Conduct one test at a time to avoid confusion when interpreting results.
- Adapt the number of variations to the volume of traffic allocated to the test to get meaningful results.
- Wait until the test reaches a statistical reliability of at least 95% before making any decisions.
- Let the test run long enough, preferably for at least a week or two and with a minimum of 5,000 visitors and 75 conversions per version, to account for differences in behavior linked to the day of the week.
- End a test if it takes too long to reach a reliability rate of 95% to avoid unnecessarily monopolizing traffic.
Measure multiple indicators, including primary and secondary objectives, such as click rate, cart addition rate, conversion rate, and average cart, to get a comprehensive analysis of results.
Take note of marketing actions during a test to avoid any external variables that may affect the results.
Segment tests to target specific groups and get more meaningful results. For example, conduct a test on new visitors rather than the current database of registered users to measure the impact of different formulations of customer advantages on a site’s registration rate.