Welcome back, everyone, to this section of the course on the normal distribution.
We've already explored probability mass functions as well as probability density functions for some
particular data distributions.
However, we have yet to explore one of the most frequently used and fundamental distributions known
as the normal distribution.
Let's explore the types of problems we're going to be able to answer once we understand the normal distribution
and identify real world data distributions that follow a normal distribution.
If you end up working with a real world data set that you can end up treating as normally distributed,
you've actually unlocked another set of tools that is equations that you can use to calculate the probabilities
of outcomes.
One of the key tools used in normal distributions is known as a Z score.
Fundamentally, once you're able to treat that real world data set as normally distributed, then you
can use the Z score and that's going to allow you to calculate the probability of data points being
between any interval range.
For example, imagine that you run a hospital.
So what you end up doing is you collect historical data around the length of human pregnancies in days.
Once you have the mean and the standard deviation of that data set and that you've tested the overall
data set to understand that it's following a normal distribution, you can calculate the probability
of a pregnancy lasting longer than any number of days.
So you can imagine that calculating that sort of probability that can aid in all sorts of tasks such
as scheduling, hospital staffing, managing inventory for deliveries or alert systems for pregnancies
lasting longer than number of days.
The key idea to keep in mind is that the answers would actually still be in probability terms.
So you would get answers in the form of something like there's a 10% chance that a pregnancy lasts longer
than a number of days.
So what are we going to cover in the section?
We'll have a discussion of mean variance and standard deviation as they pertain to the normal distribution.
Then we'll have a deeper dive into the normal distribution, otherwise known as a Gaussian distribution.
Then we'll talk about the standard normal distribution, which is a specific case of a normal distribution.
And then finally we'll talk about Z scores, which allow us to actually use a normal distribution to
calculate probabilities.
So the normal distribution is one of the most common distributions we use in business because so many
real life data sets end up resembling a normal distribution.
Formerly the normal distribution is defined as a type of continuous probability distribution for a real
value random variable.
Now, we already learned about probability density functions and the function and the general form for
a normal distribution follows this formula.
While the formula can actually be a bit complex at first sight, for example, it has E and also a PI,
which is kind of crazy.
You can take a close look at the terms and realize that in order to figure out the output f of x, you
really just need three things.
You need x itself.
And then just the mean and standard deviation for your particular data set.
You plug those in and then eventually you'll get F of X out.
So let's take a step back and think about the shape of a normal distribution and what that implies for
people who want to use its properties to answer questions.
Fundamentally, we can think of the normal distribution as a data distribution where most values tend
to fall closer to a mean value and then are distributed with some degree of variance.
For example, heights of people are normally distributed.
Most people end up being closer to the average height.
Then less people are either going to be very short or very tall.
So you'll often see a visualization of a normal distribution.
Look something like this, where most values tend to be closer to the mean, and then you have the tails
which indicate lower probabilities.
Notice how it's just a probability density function showing the likelihood of choosing a particular
data feature value.
In this particular case, you're actually seeing a normal distribution that's centered around a mean
value of zero.
However, a normal distribution can have any mean value.
So here we can see a normal distribution for the global heights of women, which tend to have a mean
value of 163 centimetres.
We've also seen the normal distribution is defined by the standard deviation, which is essentially
a metric related to the variance of the distribution.
Imagine a data set with a wider variance such as the prices used cars are sold at depending on the condition
of the car.
We could have a wide range of possible prices being paid for used vehicles, thus a larger standard
deviation value.
So here we can see a normal distribution of used car prices.
And for a second, you may be thinking, hey, this looks exactly the same as the chart you just showed
earlier for the heights of women.
However, you should notice that the range on the x axis is a lot wider.
Here we're going all the way from like 5000 to 25000, where previously, if you were to take a look
back at the other slides, notice that this is actually a much tighter standard deviation in terms of
centimeters.
So you're probably wondering how does this work in real life?
I can read all the textbooks I want and see all the pretty visualization diagrams, but in real life
you don't start with a nice probability density function curve.
Instead, you're going to have a series of real values, such as a data set of used car prices or measurements
of heights.
So you actually have to go from the real data set first and then map it to some sort of theoretical
probability density function curve.
And remember, the formula technically allows you to do that.
You just needed to know the mean and the standard deviation, and then you can plug in your x value
and start drawing out that curve.
So imagine that we're in charge of a school and are performing standardized testing.
Perhaps we're trying to analyze the probabilities of students across the nation scoring particularly
well on the test.
Can we use our single school as a sample of the overall student population?
So let's imagine that we have this data set and we have 100 students and their test scores, first student
scores, 70%, second students scored 85% and so on.
So we're going to take a real data set.
And then now we actually need to test if this data is normally distributed.
There's a couple of different ways to do this.
A simple way is to just visualize the data so I can take this data set and then actually create a histogram
and you'll notice that it starts to look like a normal distribution curve and you can play around with
the bin count to get a better idea if it's following a normal distribution.
So again, here we can see that visually speaking, it looks like it's following a normal distribution
in the histogram.
But sometimes that may not actually be as clear.
In that case, we have more stringent mathematical tests for normality.
A test for normality, basically tests.
Hey, what's the probability that this data set is actually normally distributed?
So there are several normality tests.
Here's a list of a bunch of them.
In fact, there's so many, there's actually a full Wikipedia page on the different normality tests.
I also find it curious that so many of these normality tests have two names attached to them, but that's
just something you can explore on your own.
But keep in mind you basically just pass in your data set to this test and then it reports back a probability
of being normally distributed.
So again, technically speaking, these tests do not tell you for certain whether your data set is normally
distributed.
They just usually present some sort of p value or metric or probability saying, Hey, your actual data
set has like a 99% chance of being normally distributed.
Now, more specifically, these tests typically operate using what's known as a hypothesis paradigm,
where you posit a hypothesis that your particular data sample, that is that data sample of student
test scores is normally distributed or comes from a normally distributed population.
Now, keep in mind, we recommend that you perform both a basic visual test and a normality test before
assuming you can treat a data set as normally distributed.
Doing both the normality tests and having that confirm there's a high probability that your particular
data set is normally distributed and doing the visual to see.
Yes.
And DH this looks like it's normally distributed is a really nice way of saying, okay, I can start
treating this data set as being or belonging to a normal distribution.
So let's head back to our real world example of student tests.
So let's imagine we just conducted a SHAPIRO Wilk normality test.
Then We've also visualized this particular data set.
In that case, if the SHAPIRO will test gives me a high confidence that our actual data set is normally
distributed and visually, I can see it looks like it's normally distributed.
Then what I'm going to do is treat it as normally distributed or belonging from a normal distribution
population.
So now that I can treat this data as normally distributed, that means I can use that mean and standard
deviation to begin to answer questions.
Again, recall that the PDF formula that is the probability density function that I saw earlier in the
slides that just needed the mean standard deviation in order to draw that curve.
Then you just plug in the x values and it'll draw the curves for you.
You should also keep in mind that normal distributions come with very unique properties of the mean
and standard deviation.
Essentially, 34.1% of all the data is going to fall between zero and minus one times the standard deviation.
That is to say that 68.2% of all the values are going to fall within a standard deviation around the
mean.
In this particular diagram, we're showing the mean as zero, but that doesn't have to be the case.
So since we can calculate the mean and standard deviation from our data set, we can now begin to plug
in formulas to answer questions.
These calculations are so common that a standard score or Z score system allows us to easily convert
the probability space of the normal distribution using our data set values.
And you don't need to worry about completely understanding the diagram or what I'm discussing here.
I just want to give you an overview of what you're going to learn in this section.
So the main idea is you take your real world data set, you check if it's normally distributed, then
you can apply what's known as a Z score, which again uses the mean and standard deviation to begin
to actually answer questions like What's the probability that a value falls above or below X or within
the ranges of particular X values?
The absolute value of Z represents the distance between that raw score X and the population mean in
units of the standard deviation.
This is all to say that you're going to actually be able to calculate probabilities of particular x
interval ranges.
Z is negative when the raw score is below the mean and positive when above.
And again, we're going to discuss these scores in more detail later on in the section of the course.
And this section of the course basically builds up to understanding Z scores.
Technically, we use the sample statistics and not the population statistics in this particular example.
So let's revisit our data.
Imagine you wanted to know what's the probability of a student in, let's say, the entire nation scoring
above an 80 on the test.
Now, I have my data distribution, but it's technically just a sample from my school.
Now I want to use my sample to figure out the probability of the entire population.
So again, you should notice how this differs from the question How many students in our particular
dataset scored 80 or above on the test?
That's not what I'm actually asking here.
I'm thinking in terms of how do I use this sample dataset to answer questions about the larger population?
And when you read studies on in journals or headlines and newspapers, they're not answering this question
of how many students from this particular data set.
Instead, they're using things like normal distributions and Z scores to try to figure out probabilistic
findings reflective of the larger population.
So the purpose of this exercise is to use our student sample to gain insight on the overall population
of students.
That also is going to come with some requirements like having a large enough sample.
I can't do this sort of thing with just one student.
And later on we'll discover that around 30 minimum data points are going to be needed to begin to do
things like test for normality.
So in our particular example of the students, we have a mean test score of 69.32 and a standard deviation
of 7.59.
So now I can end up doing is we'll discover that if I'm trying to answer the question, what's the probability
that a student in the general population of the nation is going to score higher than 80%?
Then I set X as 80.
Then I take that value.
I plug in x is equal to 80 mean of 69.32 and a standard deviation of 7.59.
Eventually, we'll learn about the Z score formula and we'll plug those in.
That leads to a Z score of 1.407 and then that will end up returning what's known as a P value from
the Z table.
Keep in mind, we're going to cover this in a lot more detail later on, but essentially that begins
to let you answer questions like, Hey, what's the probability that someone scores below that X value?
What's probability somebody scores above that X value?
And what's the probability that somebody scores within a particular range?
And you can see those calculations here.
And you can also just Google search for Z score calculator and you'll be able to do this sort of thing
yourself.
So for this particular example, keep in mind I just made up the data, but you would end up calculating
the mean, the standard deviation, and you pass in the particular x value you're interested in.
In this case I said 80, and then after doing this calculation with the Z score and understanding that
it's normally distributed, I can calculate that the probability of a student in the general population
scoring above an 80 on the test is approximately equal to 8%.
Again, this was for this particular made up data set.
So just as in our previous discussions of data distributions, once you've confirmed the recognized
a normal distribution in your data, you can use the Z score, relationships or formulas to easily answer
probability questions about intervals in your sample data.
The reason why the normal distribution is so critical to understand is because so many data sets in
nature happen to be normally distributed.
Part of the reason behind that has to do with something known as the Central Limit Theorem, which we're
going to discuss later on in the course.
For now, let's take a closer look at the normal distribution and its unique properties.
But again, keep in mind the main steps and the main ideas.
You take your real world data set, you visualize it and maybe perform a normality test to understand
that it's normally distributed.
From there you calculate the mean and standard deviation, and then from that you're going to end up
using what's known as a Z score to be able to answer questions with probabilistic answers.
So let's learn how to do that.
We'll see you at the next lecture.