Author: Evgeny Bodyagin, https://dsprog.pro
Introduction
It is necessary to analyze the effectiveness of the new design for the site. We have data about site users with a conversion value for each user/session. Let’s do an A-B test. Let’s call the former design form “A”; Let’s call the new design form “B”.
Experiment Design
- Null Hypothesis: No difference in performance between Design A and Design B.
- Alternative Hypothesis: There is a significant difference in performance between Design A and Design B.
- Processing Object: Two samples of site visitor sessions mixed randomly. The sample sizes are the same.
- Target Audience: Site visitors in the last 40 days.
- Metrics: Impressions, Clicks, Purchases, Earnings
- Duration of the test: 40 days
- Levels of significance:
- α = 0.05
- β = 0.2
- Campaigns run at the same time, and the audience is divided randomly.
Data structure
- Campaign Name: The name of campaign.
- Impression: The number of ad impressions.
- Click: The number of clicks on the banner and went to the site.
- Purchase: The number of times users have made a purchase.
- Earning: Earnings after purchasing goods.
import itertools
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import statsmodels.stats.api as sms
from scipy.stats import ttest_1samp, shapiro, levene, ttest_ind, mannwhitneyu, pearsonr, spearmanr, kendalltau, \
f_oneway, kruskal
# this content is from dsprog.pro
from statsmodels.stats.proportion import proportions_ztest
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', 10)
pd.set_option('display.float_format', lambda x: '%.5f' % x)
Definition of levels: test significance and test power
import scipy # Significance level (Probability of Type I error) that is used to decide whether to reject the null hypothesis. alpha = 0.05 # Power of the test (probability of type II error), which indicates the probability of not detecting a real effect when it exists. beta = 0.2
Loading data
The experiment was carried out for 40 days. Let’s load the existing data.
a_group = pd.read_csv('input/Xa_group.csv', delimiter=',')
a_group.head()
| Campaign Name | Impression | Click | Purchase | Earning | |
|---|---|---|---|---|---|
| 0 | a campaign | 77198 | 4774 | 307 | 1933 |
| 1 | a campaign | 126169 | 4285 | 574 | 2572 |
| 2 | a campaign | 114917 | 4134 | 720 | 2212 |
| 3 | a campaign | 133553 | 5144 | 545 | 1957 |
| 4 | a campaign | 90761 | 1948 | 186 | 1675 |
b_group = pd.read_csv('input/Xb_group.csv', delimiter=',')
b_group.head()
| Campaign Name | Impression | Click | Purchase | Earning | |
|---|---|---|---|---|---|
| 0 | b campaign | 115507 | 1942 | 656 | 2335 |
| 1 | b campaign | 130944 | 2844 | 606 | 2748 |
| 2 | b campaign | 175336 | 2953 | 287 | 3690 |
| 3 | b campaign | 139732 | 3550 | 709 | 2184 |
| 4 | b campaign | 108745 | 3261 | 297 | 2454 |
desired_palette = ["#0099cc", "#009900"]
data_all = pd.concat([a_group, b_group])
fig, ax = plt.subplots(ncols=4, figsize=(26,5))
ax1 = sns.barplot(data=data_all, x='Campaign Name', y='Impression', errorbar=('ci', False), ax=ax[0], estimator='sum', palette=desired_palette)
ax2 = sns.barplot(data=data_all, x='Campaign Name', y='Click', errorbar=('ci', False), ax=ax[1], estimator='sum', palette=desired_palette)
ax4 = sns.barplot(data=data_all, x='Campaign Name', y='Purchase', errorbar=('ci', False), ax=ax[2], estimator='sum', palette=desired_palette)
ax5 = sns.barplot(data=data_all, x='Campaign Name', y='Earning', errorbar=('ci', False), ax=ax[3], estimator='sum', palette=desired_palette)
# this content is from dsprog.pro
ax1.set_title('Impression')
ax2.set_title('Click')
ax4.set_title('Purchase')
ax5.set_title('Earning')
plt.show()
Roadmap for formulating hypotheses
- Assumption that the data is normally distributed (normality)
- Assumption that the dispersion is homogeneous (homogeneity).
Both assumptions will be tested through a test of statistical significance. The type of tests will be selected for each task. If both assumptions are valid, then two independent sample t-tests are applied. If one of the assumptions can be rejected, the Mann-Whitney U-test (non-parametric test) will be applied.
Note: The t-test is used to test for the existence of a statistically significant difference between the two group A and group B by looking at the means. This is a parametric test.
Shapiro-Wilk test
There are many tests for normality. The most famous of them are the criterion:
- Chi-square test;
- Kolmogorov-Smirnov criterion;
- Lilliefors criterion;
- Shapiro-Wilk normality criterion. Let’s use the last one.
Unable to reject Null Hypothesis for “Impression” in “group A”, because “Test Stat group A” = 0.9810 and “p-value_a” = 0.7250 Unable to reject Null Hypothesis for “Impression” in “group B”, because “Test Stat group B” = 0.9609 and “p-value_b” = 0.1801
Unable to reject Null Hypothesis for “Click” in “group A”, because “Test Stat group A” = 0.9882 and “p-value_a” = 0.9465 Unable to reject Null Hypothesis for “Click” in “group B”, because “Test Stat group B” = 0.9709 and “p-value_b” = 0.3851
Unable to reject Null Hypothesis for “Purchase” in “group A”, because “Test Stat group A” = 0.9775 and “p-value_a” = 0.5972
Unable to reject Null Hypothesis for “Purchase” in “group B”, because “Test Stat group B” = 0.9690 and “p-value_b” = 0.3341
Unable to reject Null Hypothesis for “Earning” in “group A”, because “Test Stat group A” = 0.9790 and “p-value_a” = 0.6524 Unable to reject Null Hypothesis for “Earning” in “group B”, because “Test Stat group B” = 0.9734 and “p-value_b” = 0.4570
So. None of the indicators: Impression, Click, Purchase cannot reject the null hypothesis about the normality of the distribution.
Levene test
We run Levene’s test to see if there are significant differences in variation between groups.
columns = ["Impression", "Click", "Purchase", "Earning"]
for column in columns:
test_stat, pvalue = levene(a_group[column], b_group[column])
if (pvalue < alpha):
print('Reject Null Hypothesis for "%s" because "Test Stat" = %.4f and "p-value" = %.4f ' % (column, test_stat, pvalue))
else:
print('Unable to reject Null Hypothesis for for "%s" because "Test Stat" = %.4f and "p-value" = %.4f ' % (column, test_stat, pvalue))
Unable to reject Null Hypothesis for for “Impression” because “Test Stat” = 0.5893 and “p-value” = 0.4450
Unable to reject Null Hypothesis for for “Click” because “Test Stat” = 0.0978 and “p-value” = 0.7553
Unable to reject Null Hypothesis for for “Purchase” because “Test Stat” = 0.0232 and “p-value” = 0.8793
Unable to reject Null Hypothesis for for “Earning” because “Test Stat” = 0.2993 and “p-value” = 0.5859
So. None of the indicators: Impression, Click, Purchase can not reject the null hypothesis of the absence of significant differences in the variations between groups.
According to the hypothesis formulation roadmap defined above, this means that we can apply two independent sample T-tests.
T test
columns = ["Impression", "Click", "Purchase", "Earning"]
for column in columns:
test_stat, pvalue = ttest_ind(a_group[column], b_group[column], equal_var=True)
if (pvalue < alpha):
# this content is from dsprog.pro
print('Reject Null Hypothesis for "%s" because "Test Stat" = %.4f and "p-value" = %.4f ' % (column, test_stat, pvalue))
else:
print('Unable to reject Null Hypothesis for for "%s" because "Test Stat" = %.4f and "p-value" = %.4f ' % (column, test_stat, pvalue))
Reject Null Hypothesis for “Impression” because “Test Stat” = -4.1173 and “p-value” = 0.0001
Reject Null Hypothesis for “Click” because “Test Stat” = 2.7901 and “p-value” = 0.0066
Unable to reject Null Hypothesis for for “Purchase” because “Test Stat” = -1.2474 and “p-value” = 0.2160
Reject Null Hypothesis for “Earning” because “Test Stat” = -7.0324 and “p-value” = 0.0000
So, using the t-test, we determined the only parameter (Purchase) that does not have a statistically significant difference between group A and group B. For the rest of the parameters, it is possible to reject the null hypothesis.
Subtotal
Considering that all indicators have a normal distribution, as well as the absence of significant differences in the variations between groups, it is possible to complete the data processing.
The conclusion is this. The new design B has an impact on performance:
- Impression,
- click,
- Earning
The new design B has no effect on the score:
- Purchase
However, it is possible to create a combined function that, taking into account normality, the significance of variations, used or did not use the Mann-Whitney U-test (non-parametric test).
Mann-Whitney U-test in data processing
columns = ["Impression", "Click", "Purchase", "Earning"]
for column in columns:
hypothesis_checker(a_group, b_group, column)
========== Impression ==========
*Normalization Check:
Shapiro Test for Control Group, Stat = 0.9810, p-value = 0.7250
Shapiro Test for Test Group, Stat = 0.9609, p-value = 0.1801
*Variance Check:
Levene Test Stat = 0.5893, p-value = 0.4450
Independent Samples T Test Stat = -4.1173, p-value = 0.0001
H0 hypothesis REJECTED, Independent Samples T Test
========== Click ==========
*Normalization Check:
Shapiro Test for Control Group, Stat = 0.9882, p-value = 0.9465
Shapiro Test for Test Group, Stat = 0.9709, p-value = 0.3851
*Variance Check:
Levene Test Stat = 0.0978, p-value = 0.7553
Independent Samples T Test Stat = 2.7901, p-value = 0.0066
H0 hypothesis REJECTED, Independent Samples T Test
========== Purchase ==========
*Normalization Check:
Shapiro Test for Control Group, Stat = 0.9775, p-value = 0.5972
Shapiro Test for Test Group, Stat = 0.9690, p-value = 0.3341
*Variance Check:
Levene Test Stat = 0.0232, p-value = 0.8793
Independent Samples T Test Stat = -1.2474, p-value = 0.2160
H0 hypothesis NOT REJECTED, Independent Samples T Test
========== Earning ==========
*Normalization Check:
Shapiro Test for Control Group, Stat = 0.9790, p-value = 0.6524
Shapiro Test for Test Group, Stat = 0.9734, p-value = 0.4570
*Variance Check:
Levene Test Stat = 0.2993, p-value = 0.5859
Independent Samples T Test Stat = -7.0324, p-value = 0.0000
H0 hypothesis REJECTED, Independent Samples T Test
The results of the combined function are no different from the previously announced intermediate conclusions. However, it has the potential to calculate cases where a non-parametric criterion is needed.
Conclusions
Both design options A and B are characterized by “Impression”, “Click”, “Purchase”, “Earning”. None of the indicators: Impression, Click, Purchase cannot reject the null hypothesis about the normality of the distribution. None of the indicators: Impression, Click, Purchase can not reject the null hypothesis of the absence of significant differences in the variations between groups. These two facts made it possible to apply the parametric T-test. As a result of this test, it was found that only one parameter (Purchase) did not have a statistically significant difference between group A and group B. Thus, there is a significant difference in efficiency between design options A and B.
Additionally, a combined function was created which, taking into account the normality, significance of variations, either uses or does not use the Mann-Whitney U-test (non-parametric test). It has the potential to calculate cases where a non-parametric criterion is needed.
=======================
For the full content of this article; for cooperation with the author, write to Telegram: cryptosensors
The full content of this article includes files/data:
- Jupyter Notebook file (ipynb),
- dataset files (csv or xlsx),
- unpublished code elements, nuances.
Distribution of materials from dsprog.pro is warmly welcome (with a link to the source).
