Onsite Analytics: Metric Definitions
- 17 Oct 2023
- 9 Minutes to read
Onsite Analytics: Metric Definitions
- Updated on 17 Oct 2023
- 9 Minutes to read
Article Summary
Share feedback
Thanks for sharing your feedback!
Onsite Analytics has compiled the metrics that help you get a better understanding of your campaigns' performance.
Summary metrics give you the unique summary of all the personalizations. The numbers can change upon any filter applied. Secondary goals will also display on this summary.
Below the Summary Metrics, you can see the variants and control groups along with variant metrics that give out the numbers on personalization level.
All these metrics are available on the A/B campaigns, 100% campaigns, and Onsite Analytics pages.
Summary Metrics
Summary Metrics, unlike Variant Metrics, give you the unique numbers of all personalizations. For instance, a user has seen 3 personalizations but has completed one transaction. While this transaction is attributed to all the personalizations this user has seen, it is counted as one single transaction in the Summary Metrics.
The summary metrics are as follows along with example cases:
- Incremental revenue: For example, personalizations of which control groups and variants had 1,000 visitors each, and the revenue of the control groups and variants is $500 and $1,125 respectively. Given the extra revenue generated by the variants, the incremental revenue of these personalizations is $625.
- Incremental conversions: Take, for example, a case where there are 1,500 conversions in the variants and 1,000 conversions in the control groups. The incremental would be 500 conversions.
- Conversion rate: Take, for example, a case where the conversion rate of the control groups is 1.0% and that of the variants is 1.5%, the conversions for each are 10 and 15 respectively.
- Conversion rate uplift: Take a case as an example, where the variants had 1,500 conversions and control groups had 1,000 conversions. The incremental conversions would be 500 conversions. In this case, the conversion rate uplift would be calculated as 25%.
| Metric | Definition | Formula |
|---|---|---|
| Incremental Revenue | Represents the total extra revenue that is generated by any Insider personalization in A/B testings. Incremental revenue is one of the important indicators to decide if a personalization is a winner. The calculation of this metric is based on the comparison between the performance of the control group and that of the variant (Insider group). | Revenue (unique) x ΣIncremental (rev) / ΣVar (rev) |
| Incremental Conversions | The extra conversions gained from the variants when compared to those from the control groups | Total Conv (unique) x (ΣIncr. conversions / ΣVar (purchases)) |
| Conversion Rate (CR) | Indicates the percentage of your visitors who complete the goal for your personalizations. Calculated separately for control and variant groups. | Average of Var (CR)= ΣVar (CR) / # of variants Average of Control groups (CR) = ΣCG (CR) / # of CG’s |
| Conversion Rate Uplift | Calculated based on the ratio of incremental conversions to the difference between the total conversions and incremental conversions | Incremental Conv / (Total conversions (unique) - Incremental Conv) |
| Average Order Value (AOV) | Calculated from unique transaction data. Meaning if user saw 3 campaigns and made a purchase, then there’ll be 1 unique purchase | Unique revenue / Unique conversions |
| Revenue from Click | Unique revenue that is generated through clicks | Unique revenue from clicks |
| Purchases from Clicks | The sum of purchases from clicks for all the variant that has this goal | Σ Variant (Purchases from clicks) - unique |
| Clicks | The sum of variant clicks | Σ Variant (Clicks) - not unique |
| Impressions | The total number of impressions is calculated by summing up the impressions of each variation. Each time a user encounters the campaign on a different device or browser, we increment the impression count by "+1," even if it's within the same session. Consider this: if a user views a campaign on Chrome desktop, Chrome mobile, Firefox mobile, and Firefox desktop, it contributes to a total of 4 impressions. Importantly, this count remains the same even if it's the same user engaging with the campaign across different devices or browsers during a single session. Refreshing the page within the same session doesn't affect the impression count, provided the user has already seen the campaign and an impression has been counted as 1. For instance, with a re-eligibility duration of 7 days, if a user views the campaign today and then sees it again tomorrow, the impressions stay at 1. However, if the user encounters the campaign once more after a lapse of 10 days, the impression count increases to 2. | Σ Variant (Impressions) - not unique |
Variant Metrics
See below for the variant metrics that give out the numbers on personalization level.
| Metric | Definition | Formula |
|---|---|---|
| Current Status | Shows the activation status of the variant (active, test, passive). Accordingly, you can sort the results. | - |
| Start Date | The day your personalization is released or set to Active in DD/MM/YYYY format. | - |
| Incremental Revenue | The extra revenue generated by the variant, calculated by finding the difference between the variant and the control group | Var(Rev) - ((Var (Imp) x CG (CR))x CG(AOV)) |
| Conversion Rate Uplift | Calculated based on the ratio of the conversion rate of the variant to that of the control group. | (Var (CR) - CG (CR)) / CG (CR) |
| Average Order Value Uplift | Calculated based on the ratio of the average order value of the variant to that of the control group. The difference between the average order value performance of the variant and control group will give this uplift. | (Var (AOV) - CG (AOV)) / CG (AOV) |
| Impressions (Imp) | The number of visitors who have been in any group of the personalization. The impressions for variant and control group are represented separately. | Number of Var impressions Number of CG impressions |
| Significance | The global statistical term for A/B testing reports. Determines whether a report is reliable to take an action depending on it. Significance depends on the size of the experiment, meaning the number of impressions, and the difference in the performance, in other words uplift. If the significance is near 0%, it means the conversion of the control group is near to that of the variant group. If the significance is near 100% and the uplift is negative, the conversion of the control group is more than that of the variant group. If the significance is near 100% and the uplift is positive, the conversion of the variant group is more than that of the control group. | Z-Value=ABS((cr_a-cr_base) / (SQRT(((sales_a+sales_base) / (imp_a+imp_base)) (1-((sales_a+sales_base) / (imp_a+imp_base)))(1/imp_base+1/imp_a)))) |
| Probability to Win | Illustrates the performance of the tested variant in achieving the selected conversion goal compared to other variants, including the control group. It serves as an indicator of the likelihood of the variant's success. The percentage is calculated using Bayesian Significance, leveraging the collected data thus far. | P(A|B) = P(B/A)P(A) / P(B) |
| Control Group Impressions | It represents the number of CG impressions. Each time a user views the campaign on a device or browser, we increase the impression count by "+1." This count is not affected by changes in the user's device or browser within the same session. For instance, if a user sees a campaign on Chrome desktop, Chrome mobile, Firefox mobile, and Firefox desktop, it would result in 4 impressions, even though it's the same user. Refreshing the page within the same session does not alter the impression count. If a user sees the campaign today and views it again tomorrow, the impressions remain at 1. However, after the campaign's re-eligibility duration of 7 days, if the user encounters the campaign again, the impressions will increase to 2. For instance, if the user sees the campaign 10 days later, the total impressions will be 2. | - |
| Variation Group Impressions | It represents the number of Var impressions. Each time a user views the campaign on a device or browser, we increase the impression count by "+1." This count is not affected by changes in the user's device or browser within the same session. For instance, if a user sees a campaign on Chrome desktop, Chrome mobile, Firefox mobile, and Firefox desktop, it would result in 4 impressions, even though it's the same user. Refreshing the page within the same session does not alter the impression count. If a user sees the campaign today and views it again tomorrow, the impressions remain at 1. However, after the campaign's re-eligibility duration of 7 days, if the user encounters the campaign again, the impressions will increase to 2. For instance, if the user sees the campaign 10 days later, the total impressions will be 2. | - |
| Incremental Conversions | The number of the extra conversions generated by any group of the personalization. It is calculated based on the comparison of the variant performance to the control group performance. | Var (Conv) - (Var (Imp) x CG (CR)) |
| Revenue (rev) | Shows the revenue gained at any group of your personalization. The revenue for variant and control group is represented separately. | Revenue gained at each group |
| Conversion Rate (CR) | Indicates the percentage of your visitors who complete the goal for any group of the personalization. It is calculated based on the ratio of the conversions to the impressions. | Control group: CG (Conv) / CG (Imp) Variant: Var (Conv) / Var (Imp) |
| Average Order Value (AOV) | Calculated based on the ratio of the total revenue gained by any group of your personalization to the conversions of that group | Control group: Var (Rev) / Var (Conv) Variant: Var (Rev) / Var (Conv) |
| Control Group Conversions | Shows the number of purchases gained at any group of your personalization. You can check below the questions to comprehend it better:
It counts as 1 conversion regardless of the number of purchased items.
If you make 2 separate purchases within the specified time (conversion period), it counts as 2 conversions. For instance, with an "add to cart" goal and adding 3 different items to the basket after logging in, it will be 1 impression but 3 conversions.
No, it counts as 1 conversion, even if the page is refreshed.
Similar to the second question, you'll observe the case of multiple conversions under a single impression. For instance, with an "add to cart" goal, adding 3 different items to the basket in one session would be 1 impression and 3 conversions. | Number of purchases gained through control group. |
| Variation Group Conversions | Shows the number of purchases gained through the variation. You can check below the questions to comprehend it better:
It counts as 1 conversion regardless of the number of purchased items.
If you make 2 separate purchases within the specified time (conversion period), it counts as 2 conversions. For instance, with an "add to cart" goal and adding 3 different items to the basket after logging in, it will be 1 impression but 3 conversions.
No, it counts as 1 conversion, even if the page is refreshed.
Similar to the second question, you'll observe the case of multiple conversions under a single impression. For instance, with an "add to cart" goal, adding 3 different items to the basket in one session would be 1 impression and 3 conversions. | Number of purchases gained through variation group. |
| Effective Days | Shows the number of days the variant had more than 10 impressions. | - |
| Clicks | Any click on a campaign except for white spaces and the close button is counted as a click. If you see a campaign and click it, it's counted as a click. If you visit another page, see the campaign again, and click it, no click is logged until you fall into the re-eligibility duration of the campaign again. | - |
| Unique Revenue | This value is derived from unique transaction data. In other words, if a user views three campaigns and makes a purchase, it results in a single unique purchase. | - |
| Unique Conversion | This value is calculated from unique transaction data. In essence, if a user views 3 campaigns and makes a purchase, it will be counted as 1 unique purchase. | - |
Was this article helpful?