Bayesian vs Frequentist Inferences in A/B Testing

A/B testing method helps you compare different versions of your web page, app, or any message you are sending to your users. These versions are called variants and each variant is delivered to a certain amount of your website’s traffic. Insider uses the Bayesian inference to calculate the probability that allocates traffic for each variant.

Bayesian or Frequentist

This article will discuss two different schools of statistical inference for probability: The Bayesian and the frequentist.

Bayesian inference considers probability as the chance of an event happening while the frequentist inference calculates the probability of an experiment to have the same outcomes under the same circumstances.

These two approaches interpret probability differently. For instance, the Bayesian approach might come up with the probability of a variant of your experiment to be better than the control group. Meanwhile, the frequentist p-value might determine whether your experiment has a high probability of having statistically significant results. P-value is the probability of a false positive, but not the probability of a specific event happening as the Bayesian approach suggests.

Let's discuss it further with an example. Suppose that you want to learn the average age of the people who live in Oceania. Before running any calculation, a frequentist would not know the average age, but know that the average is a fixed number.  

When you do not have access to the age data of every single person in Oceania, you cannot make the exact calculation. So you take a sample group of 10,000 people, take the average of their age and come up with a point estimation. While the average age would be an unknown number to a frequentist, it would be an unknown distribution to a Bayesian as there would be more variables to interpret.

Bayesian Probability

When the inference starts to take the age data of 10,000 Oceania residents, it distributes each age around the average age that is calculated so far. You can imagine this distribution as a bell-shaped curve with the average being in the middle and the ages being placed around it. When you add more data to the distribution, you can see that the bell is having a pointier end.

Before coming up with a point estimation, the Bayesian inference might incorporate some assumptions about the probability such as the range of the average age (e.g. between 25-32). In the Bayesian approach, the concept of probability is associated with prior information about an event.

In a similar way, a Bayesian would find the probability of a conversion rate of an ecommerce website being in a range consequential while a frequentist would base their inference on a fixed number and find this probability inconsequential. This means that the Bayesian approach would calculate probabilities based on previous data to come up with the conversion rate. However, the frequentist approach would take the number of conversions over the number of audience.

Determination of a Winner

When calculating the probability of variants of your campaign, first you distribute your data. Then, you take a large sample to determine a winner and observe how many times each variant has higher probability. Accordingly, you can conclude that one of the variants has a probability of being better than the other one.

Calculation of Bayesian Inference

Insider uses the Bayesian model to calculate the probabilities in A/B testing. You can see below for its calculation formula.

# fix seed
set.seed(1)
# group 1
n1 = 381
m1 = 4717
a1 = 1 + n1 # posterior parameters of the beta distribution
b1 = 1 + (m1 - n1)
# group 2
n2 = 328
m2 = 4733
a2 = 1 + n2 # posterior parameters of the beta distribution
b2 = 1 + (m2 - n2)
# group 3
n3 = 10
m3 = 90
a3 = 1 + n3 # posterior parameters of the beta distribution
b3 = 1 + (m3 - n3)
# samples from a beta distribution
r1 = rbeta(1e4, a1, b1) 
r2 = rbeta(1e4, a2, b2)
r3 = rbeta(1e4, a3, b3)
r = cbind(r1, r2, r3)
# compute max group for each sample
max_group = rep(0, nrow(r)) # stores the group with maximum sample value
for (i in 1:nrow(r)) {
  max_group[i] = which.max(r[i,]) 
}
# compute probabilities of each group being the max
for (j in 1:ncol(r)) {
  prob_j = mean(max_group == j)
  print(paste0("Probability of group ", j, " being the maximum equals ", prob_j))
}