Welcome back, everyone, to this section of the course on hypothesis testing.
One of the most crucial capabilities an organization needs is the ability to test a theory or hypothesis.
Companies often want to test potential effects before rolling out new features or services to all their
users.
In this section, we're going to be talking about topics related to hypothesis testing like significance
level and type one versus type two errors, one tailed test and two tailed tests, the P value and a
B testing.
Understanding hypothesis testing provides a wealth study, statistical process and foundation for testing
new features or analyzing results as a company or organization.
Let me give you a few examples of the types of things companies can test with hypothesis testing.
So you can test things like the effectiveness of vaccines or medications.
You can also do a B testing for changes on a website such as testing whether or not a new color button
actually improves conversions.
You can also do things like testing, changes to schedules for efficiency, such as our workers just
working four days a week, more productive and workers working five days a week.
You can also do things like test the effects of different agricultural procedures on crops, like whether
or not to use fertilizer.
In general, a hypothesis can be stated as something in this term.
If we do blink, then blink will happen.
And let's get actually a little more specific and frame this in statistical terms when we're thinking
of hypothesis testing and what a hypothesis is.
We're typically thinking of something in the terms of if we do blink to an independent variable, then
blink will happen to a dependent variable.
So let's actually match up this sort of general statistical definition to some of the example cases
we've mentioned.
So, for example, I could say if we give patients a medication, then their white blood cell count
will increase.
And that could be an example of a hypothesis that I can test.
Also note that I could technically model this as a before versus after medication test so I could take
the same group of people, count their white blood cells, then given the medication and count the white
blood cells, post medication.
Or I could split people into two groups, a control group that doesn't take the medication versus a
group that does take the medication.
So let's do another example of, again, blank to an independent variable than blank will happen to
a dependent variable.
So we can say something like, If I add a fertilizer to the soil, then crop yields will increase.
So again, this test would require a fair comparison between a non fertilized crop against the same
fertilized crop.
So I should try to make sure that all the other independent variables are as similar as possible between
the groups.
So an obvious one would be the crop itself.
I probably want to compare non fertilized corn versus the same corn just fertilized.
It's not really helpful to do something like non fertilized oranges versus fertilized cucumbers.
We want to try to keep all the other independent variables the same and we'll discuss in a second about
testing multiple independent variables.
So for example, if we think of another hypothesis, I could say something like if we change the purchase
button to a larger font and brighter color, then more customers will complete a purchase.
Now you should notice this hypothesis is a little different than the ones we previously saw.
This is basically an AB test of a website with a being the original button and B being the new button
with a larger front and brighter color.
However, this particular example has an important factor.
You may have noticed that we're actually changing two independent variables at the same time.
I'm changing both larger font and brighter color.
So while it's certainly possible to conduct a hypothesis test with multiple independent variable changes
against a dependent variable, we can see that this could make it unclear what independent variable
is actually causing the effect or change in behaviour.
And statistically speaking, while there are techniques to actually conduct changes across multiple
independent variables, the cost, so to speak, is usually have to gather a lot more data.
And we've previously mentioned that companies with a lot of users like Facebook or Google typically
have enough users that testing multiple independent variable changes is actually not a big deal for
them because they can easily test small percentages of their users but still have really large data
sets like a million users out of 2 billion customers.
So again, to frame that last hypothesis, you're going to be asking yourself questions like was it
the color of the button that caused the effect or the font size or both changes put together.
And as I mentioned, there's many established statistical methods to test changes across multiple independent
variables.
But for right now, I want to focus this section on just testing singular changes to a single independent
variable.
Like many of the other topics we've covered, hypothesis testing can seem intimidating due to terminology,
and throughout this entire course, we've been trying to simplify a lot of the terminology to give you
an intuition of what's actually going on behind the scenes.
So you may hear terminology like null hypothesis and some esoteric phrasing like failed to reject the
null hypothesis.
Let's guide you through a simple example so you get more comfortable with terms like fail to reject
and null hypothesis.
So we're going to start off with a test situation or scenario.
Imagine that we're in charge of a large e-commerce company and we run a website.
And what we're going to do is we're going to change the size of the font to a larger size across the
entire website.
So all the font you see on the website is going to be slightly larger.
And we have a hypothesis that if we're going to change the font size, then the customers are going
to spend more money.
Since our website is online, we can show some customers the original font size and measure their spend
and simultaneously show another segment of customers the larger font size and measure their spend.
When conducting tests like this one.
We always want to try to make sure that the test audiences are similar to each other.
So I want to make sure my control audience that's going to see the original font size is roughly similar
to the test audience that is seeing the larger font size in order to prevent possible outside factors
from contributing to any change in the dependent variable, which in this case is spend.
So now that we understand the scenario, let's frame this so that we slowly build up to the definition
of a null hypothesis.
So I'm going to describe a situation where kind of a silly but very specific hypothesis is positive,
and then we're going to use that to build our understanding of rejecting and failing to reject a null
hypothesis.
So let's imagine that I'm going to build up to a kind of hyper specific hypothesis.
I'm going to say customers viewing larger font size will spend 100 or more dollars on the website.
And how did I come up with this hypothesis?
Maybe I ran a really small preliminary test comparing a test group to a control group.
I remember the test group is going to see that larger font size, so I run this small preliminary test.
And I have a test group.
There's three customers in it, and then a control group, another three customers.
And you probably need more than these customers.
But we're keeping things simple for now.
I take the average values of their spend and I see that on average, the test group viewing the larger
font size is going to spend 100 more dollars versus the control group.
So I come up from this preliminary testing with this very specific hypothesis that customers are viewing.
The larger font size will spend 100 more dollars on the website, and we'll see that this causes issues
further down the line.
But let's just get this idea of reject versus fail to reject, and then we'll build up to a null hypothesis,
which is a lot broader and a lot easier to use.
So again, there's 100 more dollars on average spent versus the test versus control.
So now that I've done the preliminary test, I start rolling out this experimentation to the real world.
So I test this on new customers on the website after my little preliminary test that helped me define
my hypothesis.
But then I have issues.
I noticed that when I actually roll this out to the real world, my test group is spending less than
the control group.
And now I see in pretty much every situation that the test group that is viewing the larger font size
ends up spending less than the control group.
So what happens if the rest of the experimental tests end up showing the opposite effect?
Well, in that case, I'm going to decide to reject that hypothesis, since none of my other experiments
actually ended up supporting this.
So I'm going to reject the hypothesis that customers viewing larger font size will spend 100 or more
dollars on the website.
Now let's imagine that we rewind the time.
So now again, I'm going to test the new font size on the web page for a variety of groups.
And this time, if we rewind time and pretend that we were to redo this experiment, what happens if
I get the same trend?
But it's not exactly $100 difference?
So I test this out in a variety of experiments, and the test group does in fact spend more than the
control group.
But you'll notice if you pay close attention to these actual experimental examples, it's not exactly
a $100 difference.
So in one experiment it was 103.
Another experiment is 160, 75 and 72 difference and so on.
So here we have a bit of a conundrum.
None of the experiments are exactly reflecting my hypothesis, so I cannot say that my hypothesis is
exactly 100% correct because remember, my hypothesis is actually really specific.
It says that the customers are going to spend 100 more dollars on the website.
And technically speaking, while the general trend was the same through the rest of my experimentation,
none of the actual experiments showed the test group spending exactly $100 more.
So the best that I conclude here is I'm going to fail to reject the hypothesis.
Notice how that wording is different than saying that the hypothesis is 100% true, or that I accept
the hypothesis as a truth.
Instead, I'm just failing to reject the hypothesis.
Since the amounts weren't exactly $100, I can't say the hypothesis is accepted.
So instead I use that very specific terminology of failing to reject the hypothesis.
So again, the difference is you reject the hypothesis or you fail to reject the hypothesis.
Now, I've already mentioned a couple of times that this idea of exactly $104 is kind of troublesome.
So is there a better way to actually define the hypothesis?
Perhaps.
Well, so far we've realized I can create the hypothesis and I can reject it.
If the testing data and the experiments don't support the hypothesis.
This was the case when testing showed that the font size change that it caused more spend than the control
groups.
We've also seen the cases that the experiments have the same trend, but not the same exact amount of
100 more dollars in spend.
I technically can't say that the hypothesis is absolutely correct.
So instead I frame my phrasing as I fail to reject the hypothesis.
So recall the hypothesis itself.
Customers viewing the larger font size will spend 100 or more dollars on the website.
Technically speaking, our goal is not really to show that the change is exactly $100.
But if I think about this from a broader viewpoint of the company or organization, what I'm really
trying to prove is that changing the font size has some effect on spend.
Now, since we've established that we operate in a framework where we either reject or fail to reject
a hypothesis, in this framework, it probably makes more sense to use what's known as a null hypothesis,
where we posit simply that there is no difference between the two groups.
That is the test and the control.
So that is there to say there is no effect or no effect.
That's why it's called the null hypothesis and you'll realize it becomes a lot easier, statistically
speaking, to just frame this null hypothesis regardless of what you're actually testing.
A null hypothesis allows us to not need to worry about an exact quantitative value of change or effect.
So don't really need to worry about saying it's going to be exactly $100 or exactly $101.
Instead, I reframed the entire question, and now I'm really just asking, did this particular change
have an effect?
And notice how that matches a lot more to our generalized statement of hypothesis.
So, for example, I could say larger font size has no effect on customer spend.
That's the null hypothesis version of the hypothesis that we saw previously.
Notice again how a null hypothesis no longer even requires us to do any preliminary tests because I
don't need to acquire some range like $100.
So this whole idea of trying to do a little preliminary test to figure out the expected range, I throw
that out the window if the null hypothesis, because all I'm doing is I'm just framing it as if this
change is going to have no effect.
That is my hypothesis.
And that null hypothesis works a lot better in the framework of reject or fail to reject.
So now let's go through a simple example of using the null hypothesis framework.
Remember, now my null hypothesis is just saying whatever this change is doesn't have an effect.
It has null effect, which means I no longer need to worry about running a preliminary experiment to
figure out something like a $100 value.
Instead, now that I've established a null hypothesis, I just go and run the experiment.
So let's imagine that we do see a difference between test and control.
Then what I can do here is I can reject the null hypothesis.
Now, in the case where test control, we're actually really quite similar and it looked like there
wasn't an effect, then I could say that I fail to reject the null hypothesis.
Now, we haven't really discussed how much of an effect is needed to determine whether we reject or
fail to reject.
That's actually going to come later on in this section with a discussion of P values and statistical
significance.
So there is going to be some metric to understand whether you decide to reject or fail to reject, but
we'll talk about that later on.
Really, the purpose of this lecture was to get you familiar with the idea of a null hypothesis and
the framework of rejecting or failing to reject.
So along the null hypothesis like larger font size has no effect on customer spend.
We also have an alternative hypothesis, which is essentially the opposite.
It basically says that a larger font size does have an effect on customer spend.
So you have the null hypothesis where there is no effect.
The alternative is that there is an effect.
Notice how neither the null hypothesis or the alternative hypothesis is actually specific to the strength
of the effect.
It's really just there is no effect or there is an effect.
Now to fully understand hypothesis testing.
Beyond this simple example, we first need to understand to learn how to evaluate the differences between
results of groups to determine whether or not the effect is what's known as statistically significant.
So let's move on to learn about how to use things like one tailed and two tailed test to set up hypothesis
testing for different situations and discover how to use p values to quantitatively state the likelihoods
of an effect being statistically significant.
We'll see you at the next lecture.