In this video.
All we want to do is relate the hypothesis testing process we've just talked through to the AB testing
process because AB testing is something we very often do in the real world.
For instance, one of the most classic examples of AB testing is the example where we test two web page
versions against each other.
In other words, we're trying to determine whether a new version of the web page converts better than
the original version.
Version A So version A is what we're currently displaying on the site.
If you visit the Web page, you'll see that version now.
And we want to know whether version B, this new version, this new variation that we've created will
result in a more favorable outcome than the outcome that's currently being generated by version A.
So to be more specific, maybe variation A of this web page has a buy now button that is blue, whereas
in variation B we change the color of that buy now button from blue to green, and we want to know if
changing the button color makes version B perform better.
Of course, this is a really simple example, but we could also run tests where we change the color
of any part of the page or we change the font or the font size.
We could change what the text actually says on the website.
Maybe in version A this button says Buy now and in version B, this button says, Get yours.
We could change the information that's being displayed on any part of the page.
We could add or remove whole sections of the web page.
Really, anything that we could imagine.
And of course, we don't have to limit ourselves to just a single web page.
We can apply this idea of AB testing to an infinite number of real world applications.
So to take something totally different, let's say maybe that we run a transcription company in which
our employees transcribe videos that are sent in by our customers and maybe our company employs thousands
and thousands of people and we want to know whether lowering the temperature in the office by one degree
will increase productivity as measured by the number of transcriptions that our employees finish each
day.
But no matter what we're testing, it's common to only make.
One change at a time, because if we make multiple changes at once.
Of course we don't know what effect that will have on the results of our AB test.
For instance, in this web page example, if we change the colour of the button and the text of the
button at the same time and we find a statistically significant result, we don't know if that statistical
significance is the result of changing the color or changing the font or both.
The combination of both.
When we make just one change at a time, then we know that the change we made was the thing that actually
had an effect.
Or we know that that change didn't result in a statistically significant difference.
Now we said that this AB testing process follows the same kind of hypothesis testing process that we've
been talking about, which means that when we run an AB test, just like with hypothesis testing, we
always start with a pair of hypothesis statements.
So that step one and when we're running an AB test, the alternative hypothesis is that the new variation,
this new variation B always leads to better results, whereas the null hypothesis is that the variation
doesn't beat the status quo.
Basically, whatever version we have now, version A when we create this new version B and we test it,
it doesn't do any better than version A, which means that they have comparable results.
The results don't change.
So this no hypothesis is that the results don't change.
Variation B doesn't perform any better than variation A whereas our alternative hypothesis says that
yes, variation B does perform better than variation A, which means that they are not equal.
Now realize here that we have used means, so the mean of variation a not equal to the mean of variation.
B for example, we'll use the mean when the result that we're trying to improve is something like maybe
average revenue per share, right?
If people are looking at this web page and then clicking on this button to make a purchase, maybe we're
trying to measure the amount of money that they spend, in which case we could use a mean because we
could look at the mean spend when our customers see variation A versus the mean spend when our customers
see variation B, And so we would use mean because our alternative hypothesis might say that the average
revenue per user for variation A is not equal to the average revenue per user for variation B, those
two means are different.
That would be an example of continuous data.
Remember previously that we talked about continuous versus discrete data.
Well, mean revenue.
If revenue can be really any amount more or less than revenue would be an example of continuous data.
But we can also have discrete data like the simple.
Yes, no question Did they click this button or not?
In which case we would talk about the proportion of customers who clicked the button when they were
looking at variation A versus the proportion of customers who clicked the button when they were looking
at variation B And in that case we would write our null and alternative hypothesis statements this way,
where the alternative hypothesis says that the two proportions are not equal, whereas the null hypothesis
says that the two proportions are equal.
Changing to variation B doesn't affect the proportion of customers who click the button.
The proportion remains the same in variation B as it was in variation A The two proportions are equal
in that null hypothesis.
Realize here also that regardless of which kind of hypothesis statements we're looking at, in both
cases, we're running a two tailed test.
We can see that from the fact that our alternative hypothesis is not equal to and our null hypothesis
is equal to in both sets of hypothesis statements.
We have that non directional test compared to a one tailed test or a directional test where we say in
the alternative hypothesis that the mean of a is greater than the mean of B or less than the mean of
B, or that the proportion for A is greater than the proportion for B or less than the proportion for
B.
And if you think about it, it makes sense that we're running a two tailed test when we're a B testing
because not only is a two tailed test more conservative, as we've talked about in the past, but also
intuitively by creating variation B here we're saying that we don't know which variation is going to
perform better.
We don't know if A's going to be better or B's going to be better or they're going to be exactly equal.
So we don't have an idea about the directionality of A versus B, And so because we don't suspect that
direction, we have no idea whether variation B will perform better or worse.
We need to run a non directional two tailed test.
And so the alternative hypothesis is always a not equal to B, and the null hypothesis is always a equals
B.
So of course that means that for the purposes of an AB test, the null hypothesis automatically assumes
that variation B is not a meaningful improvement over variation A and running the A B test is either
going to disprove that null by showing that variation B is better and better in a statistically significant
way.
Or we're going to fail to show that B is better and in that.
Case, we would fail to disprove the null hypothesis.
Now, just like with hypothesis testing, our next step after we pick hypothesis statements is to choose
a confidence level and therefore the alpha value or the level of significance.
And a 95% confidence level is typically an industry standard.
Again, we can pick technically any confidence level we want, but it's very common to pick a 95% confidence
level and therefore an alpha value of 5% or a level of significance of 5%.
So just like with hypothesis testing, these are values that we set ahead of time.
And it's also really important when we're running an AB test to choose ahead of time, sample size and
a time interval.
So the sample size idea really isn't new in hypothesis testing.
We would pick a sample size ahead of time and we do the same thing here.
Pick a sample size, but for a B testing.
It's also extremely important that we pick a time interval ahead of time, an interval of time over
which we will run the AB test or conduct the AB test.
And it's really important that we stick to that time interval and not end the test prematurely.
The reason that setting a time interval ahead of time is so important is because AB testing tools often
won't wait for a specific amount of time before returning a result.
Instead, they'll start indicating right away as data is being collected, whether the result is showing
significance.
So imagine going back to our web page example that buying behavior on our website changes drastically
throughout a single week, or maybe even a single month.
For instance, maybe purchase volume is much higher during the week and lower on the weekends, for
example.
Maybe it's also the case for whatever reason, based on the kind of website we're running here, that
most purchases on the site are made closer to the end of a month, as opposed to maybe in the first
half of the month or something like that.
If we pick a time interval that's too short or we don't allow the test to run through the full time
interval, then we'll miss capturing data across the entire week or across the entire month, and we
might start to see that the test is looking significant for maybe just weekday buying behavior.
But we haven't let the test run through the weekend and maybe letting it run through the weekend would
actually bring the results out of statistical significance.
Or maybe if we only ran the test through the first half of the month instead of through the entire month,
when most buying is done near the end of the month, maybe the test wouldn't look statistically significant
for the first half of the month.
But if we wait an entire month now, all of a sudden when we capture all that data at the end of the
month, the test suddenly becomes statistically significant.
So when we're a B testing, we need to pick a time interval that makes sense, and then we need to stick
to that time interval without ending the test early.
Another reason for doing this is this idea here of novelty effect.
This idea of novelty effect relates to our existing customers.
If we have regular customers who visit our site over and over and over again, and we've been showing
them variation A for a long period of time, they're very used to seeing this blue button here on the
site.
If we suddenly change the color of the button from blue to green and we show them the green button,
there's this novelty effect where they just notice that difference or they notice that something's different.
And that may be creates a change in their behavior simply because we changed something.
But maybe it's not actually the case that the green button does better with new customers than the blue
button.
It's only doing better with existing customers because of this novelty effect.
And maybe once our existing customers get used to this green button, given enough time, then it won't
actually be the case that this green button performs any better than the blue button.
So we need to set this time interval so that we make sure to capture a sufficient amount of data, especially
if our data varies over time, like we talked about days of the week or throughout the month, maybe
even an entire buying season.
And we need to be able to get past this novelty effect.
So with all that in mind, we pick a competence level and therefore an alpha value.
We set a certain sample size and a time interval and we've committed to sticking to that time interval.
Our next step is going to be to calculate a test statistic.
But before we go there, let's talk about why we're even doing this a B testing in the first place.
Well, if we go back to our Web page example, maybe we get a million visitors per month to our website.
If we just decide today to change this button from blue to green, even if sales revenue or click through
rate improves in variation B here, we can't necessarily conclude that the green button is better because
maybe up until this point we've been running this web page with the blue button and we've been seeing
a certain.
Level of revenue or a certain click through rate, and then all of a sudden we change the button to
green and let's say that sales revenue increases with that change.
Well, we don't know if it increased because we change the color of the button or if it increased maybe
because the month in which we changed to this version of the page is maybe a higher volume month for
revenue in general.
Maybe we started showing this green button closer to holiday season and so revenue would have gone up
anyway.
And in fact, maybe keeping the blue button would have made revenue even higher than the revenue we
got when we changed to the green button.
So if we just make the change from variation A to variation B, we just decide overnight to make the
change.
Instead of doing any testing.
We don't really have any insight into whether the change we made is actually a good change or not.
We don't know if it's making a positive impact, a negative impact or no impact.
So ideally we'd prefer to test instead of just make the change.
And then in addition, there are often risks or even costs involved in making a change.
And so we might want to run a test before we incur all that risk or incur all of that cost to change
from A to B.
So if we have millions of customers coming to our website every month and maybe our revenue is in the
many millions of dollars, making a change without doing any testing potentially puts that revenue at
risk.
And so it's important that we're cautious about any changes that we make.
And doing some A B testing helps us make sure that we're only making changes that the data tell us will
be positive to our bottom line.
Or we talked about the example where we want to see if lowering the temperature of our office space
increases worker productivity.
Well, lowering the temperature for thousands and thousands of employees might cause our utility bills
to increase significantly.
And so maybe we want to run an AB test to look at productivity before we spend tens of thousands of
dollars or even hundreds of thousands of dollars on extra air conditioning.
So those are just some examples of why we do a B testing in the first place.
And because we're doing a B testing, that means that we're doing inferential statistics because we're
taking a sample and then using statistics about the sample to make inferences about the population.
So we are running web page variation B against a sample of our customers.
We're running web page variation A against another sample of customers, and we're comparing those samples
or we're lowering the temperature in our office space for employees in some departments and keeping
the temperature the same in other departments.
So inherently we have this idea of sampling.
So these are the parallels between hypothesis testing and a B testing.
And at this point, once we get to calculating the test statistic, what we really need to say is that
this whole A, B testing process these days is done in A, B testing software or with a B testing tools.
And that software, those tools are going to help us with every step of this process by determining
which kind of test we're going to run.
And picking a confidence level and a sample size will be able to input all of this information.
And when it comes to calculating the test statistic, there's a different test statistic for every different
kind of data and every different kind of test.
And so our goal here is not to understand all of the fine detail behind every type of test, but rather
just to get a solid enough foundation so that we understand generally what our AB testing software is
doing and what it's telling us so that we can intelligently interpret the results that we're seeing.
So when it comes to calculating the test statistic, we want to keep in mind that we're going to use
a different test statistic depending on whether we have discrete or continuous data.
We talked about this when we talked about hypothesis statements.
If we have discrete data like do our customers click the button?
Yes or no, we can think about examples of that, like click through rate, like the number of products
that are purchased.
If we just have a count of products being purchased and we can contrast that with continuous data where
of course we might have something like what we talked about earlier, the mean revenue per customer.
So you certainly want to have an idea whether your data is discrete or continuous based on what it is
you're trying to measure and improve.
And then depending on the kind of data or the kind of test will use different test statistics.
So for example, if we're testing click through rate will probably use what's called a Fisher's exact
test.
If we're looking at the number of products purchased will very likely use a chi square test and we'll
talk about Chi square later when we look at regression, if we're looking at revenue per customer,
we might use a Welch's T test or maybe even just a student's T test.
And of course we'll use a different test statistic formula for each of these different kinds of tests.
But that's what a B testing software will help us with.
If we feed it the correct information, it'll.
Help us choose the correct test statistic.
And then, of course, our hypothesis testing software helps us with the very last step, which is to
draw a conclusion by answering the question, Is the test statistic that we calculated or that the software
helped us calculate significant enough to reject the null hypothesis, in other words, to reject the
idea that variation be whatever it was, had no effect on the outcome we were looking for, had no effect
on mean revenue, had no effect on click through rate or the number of products purchased.
If we fail to reject the null.
If the test statistic is not significant enough, then we can't say that variation B is better than
variation A There's no change to the status quo.
We can't prove that there's a significant difference.
But if the test statistic is significant enough to allow us to reject the null hypothesis, then we
reject the null lending support to our alternative hypothesis, which means that we have at least some
support for the idea that Variation B performs better than variation A Now remember, just like with
hypothesis testing, the whole idea here is not about whether we found a better sample mean with variation
B than with variation A or a better sample proportion, like with the set of hypothesis statements,
a better sample proportion with variation B than with variation A Just like with hypothesis testing,
we might find a better result with variation B than with variation A But the question is, is it better
enough?
We need it to be statistically significant in order to support the idea that we think variation B will
do better across the entire population, not just the sample, then variation A So to take a super simple
example, let's say we show variation A of our web page to 1000 customers.
So 1000 customers see the blue button and let's say 1000 customers see the green button variation.
B And let's say that mean revenue per customer on the blue button page is $212 and mean revenue per
customer on the green button page is, let's say, $220.
Well, you can probably get an intuitive sense here that while there is a difference between 212 and
220 and it looks like Variation B produces more revenue than variation A, the difference is, at least
in this super simple example, the difference doesn't seem significant enough to make us feel very confident,
much less 95% confident that variation B is definitely better than variation A Whereas if you saw data
where for the 1000 customers that see the blue button mean revenue per customer is 212 but mean revenue
for the 1000 customers who see the green button is.
$500.
Just at a basic level, that's going to catch your attention way more when the difference here is almost
$300 versus $8.
So even though in both cases variation B is better than variation A, in this second case, variation
B has a much better chance of being statistically significantly better than variation A And that's the
whole point.
It can't just be better.
It has to be better enough.
It has to be statistically significant in order to feel confident at a 95% confidence level that the
improvement that we're seeing in variation B here is not contained to just the sample that that improvement
is actually going to map to the entire population.
So hopefully that gives you an idea of the foundational ideas behind this, A B testing process and
how they're so closely related to the hypothesis testing process that we've just worked through.
In the next section, we'll start talking about regression, which is all about trends in our data.
And as mentioned earlier, we'll look a little more at this test statistic we mentioned earlier, which
is the chi squared test statistic.