Fundamentals of A/B Testing

A/B testing is an experiment that compares one or more alternative versions (variants) of a page against the original, gauging their efficiency at accomplishing selected goals (visitor conversions). It helps marketers understand the performance of different versions of a piece of marketing.

The measured conversions are based on visitor actions that can range from simple conversions (e.g. a banner click, a form submit, etc.) to complex conversions (e.g. placing an item into the shopping cart, performing a full purchase, etc.).

How does it work?

An A/B test must essentially start with formulating a hypothesis that defines the elements that need to be improved and lays out how the changes will positively affect your desired outcome.

During a test, all page variants receive roughly the same visitor traffic and compete with each other for the highest conversion number or weighted value. 

When a user navigates to a page with a running A/B test on a live website, the system assigns a randomly selected variant from the given A/B test variant pool to this user. During the test duration, each page variant should receive roughly the same amount of visitor traffic.

How do you manage the allocation?

You can assign users to the variants in two ways: Manual allocation and automatic allocation. While web templates use manual allocation, Architect and Email use automatic allocation.

With manual allocation — the classic A/B testing approach—, the traffic is evenly split between the variants until a single winner variant is ultimately declared with high confidence level. Each user is randomly assigned to the variants according to the traffic allocation.

With automatic allocation (a.k.a. dynamic allocation), the highest-performing variant is dynamically observed over time and gradually served to a larger percentage of visitors as more data is collected. 

Given the effectiveness, Insider has implemented the manual allocation method for the traffic distribution.

How do you ensure randomness?

As the essence of any A/B test, random sampling means that any visitor to your website has the same probability to be chosen to see a variant of your A/B test.

Insider ensures automatically split traffic to your variants so that each variant gets a random sampling of visitors.

To ensure the highest possible effectiveness and performance, Insider utilizes the reservoir sampling algorithm, specifically the weighted random sampling. 

Weighted random sampling algorithm employs a simple random sample, without replacement, of "k" items from a population of unknown size "n" in a single pass over the items. The size of the population "n" is not known to the algorithm and is typically too large for all "n" items to fit into the main memory. The population is revealed to the algorithm over time, and the algorithm cannot look back at the previous items.

How do you ensure consistency?

Insider stores a special local storage containing the identifier of the selected variant in the visitor's browser. This local storage allows the system to maintain consistent appearance of the tested page for returning visitors, and also ensures that all conversions performed by A/B tested users are logged for the assigned variant. Local storage determines not to show the campaign repeatedly.

Best Practices

In order to get the most out of your A/B testing, we highly suggest you:

• choose the most suitable wording in your campaign based on your hypothesis so that you can appeal to your users in the right manner.
• choose the correct Call-to-Action (CTA) button in your campaign based on your hypothesis so that you can drive more conversion in an interactive way.
• ensure the links lead to the right page based on your hypothesis so that you you can direct your users as desired.
• choose the most appropriate visuals in your campaign based on your hypothesis so that you can grab your users interest more in an eye catchy way.
• append your inline personalization in the most correct way so that it will not break any functionality on your website and look harmonious together with the other elements of your website.
• check the landscape of your campaign on Mobile Experience so that it will fit perfectly.