Welcome back, everyone, to this next to the course on sampling.
When working with data, we often are limited in our ability to collect all the data in existence.
More likely and more realistically, we're really just collecting a sample from a larger population,
and we hope to use that sample to estimate characteristics of the whole population.
Understanding sampling well is critical for testing out potential business decisions or strategies.
As I mentioned, it's unlikely that we're ever going to have access to an entire population of data.
So in setting up ideas to test, we need to understand how sampling plays a role in our experimentation
process.
Now, a quick important note on this idea of sampling versus the entire population.
It should be noted that certain modern technology companies with a billion plus user bases like Gmail
or Facebook or Instagram or these other services, they actually contain a strategic advantage due to
their scale of being able to conduct tests on enormous samples while only affecting a small percentage
of users.
So you can imagine that if you have a billion users or 2 billion users on a service such as Facebook,
that if you want to quickly conduct a sample test, so to speak, on a million users, that is much
larger of a sample than most other companies have.
However, with the scale of something like Facebook, it's actually a very small percentage of their
users.
And this combine of statistics is actually a strategic advantage, and I want you to keep that in the
back of your mind as we continue to discuss sampling.
So in this section of the course, we're going to be discussing sampling and bias, the central limit
theorem, the student's T distribution and confidence intervals.
I want to begin by exploring some ideas of sampling tests and how sampling relates to the central limit
theorem, which is a crucial concept.
We can think of sampling as grabbing data instances from a larger data distribution.
And let's actually frame this in a more realistic case.
Imagine we are testing delivery via air drones and we're tracking how off target the packages are.
So we take these drones and they happen to, while they're flying in the air, drop a package onto the
ground and you can actually look this up on YouTube and see videos of startups doing this exact task.
So our company, however, is simply a logistics company.
We don't actually want to manufacture the drones.
Instead, what we're going to be doing is running some experiments.
We want to test different brands of drones to understand their performance, more specifically, understand
their accuracy in their ability to drop a product or box on target.
So the purpose of testing the drones is so that when we go into production, we can have some idea of
how accurate different brands of drones are in their delivery targets.
So what we could do is we could begin to think of our testing of the drones we actually have delivered
on hand as a sample of larger future population of all drone deliveries.
Because remember, realistically, we're not going to be able to test every single drone that we're
ever going to buy off the assembly line.
Instead, we are just working off a sample of drones that we hope is going to be representative of all
the drones that come off the assembly line.
So let's imagine that we're tracking the measurements of how far off the packages are and we end up
creating a sample distribution.
So what we end up doing here is we run a bunch of tests, take some drones, they have a product, and
then they drop it onto the ground and we have a particular target that they need to meet.
And what we're doing is we're going to measure the distance of how off they are from the target.
This is made up data, but you get the idea.
We're just measuring, hey, this particular drop was off by 100 centimeters, this next one was off
by 110 centimeters, etc., etc..
And for each of those, I'm going to draw a little point and I could then further organize that data
into a histogram.
And that essentially creates a data distribution from real life experimentation.
So I'm essentially now just counting how many drops were 100 centimetres to 150 centimeters off target
or something like that.
Depending on how you define your bins, you can see it looks like I'm starting to build out a data distribution
from my data of the drones.
Now, we can also see that in this particular example, if I were to randomly choose a single data point,
it would likely be close to 100.
So far, all these concepts that we've discussed, we've already learned about when talking about data
distributions.
But now I want to present two different scenarios and how the knowledge learned in this section can
help us.
Basically, giving you some motivation and framing of why sampling is so important.
So let's imagine this first situation or scenario one.
Imagine we had some partners across the globe conducting their own tests and they're creating their
own data distribution for another drone.
But now we have a problem.
They forgot to write down the actual brand name.
So they have all this data, but they're actually not sure what the brand of drone was.
So using the idea of sampling from data distributions, would there actually be a way, statistically
speaking, to see if the data sets were likely to come from the same population that is the same brand
of drone?
Well, later on, we're going to learn that we can actually begin to treat this situation as a test,
which is going to allow us to test to see if the sampling from both drones were likely to come from
the same population.
And in this particular case, that's going to be based off the results of the data distribution of dropping
off from the target location.
Do these drones actually show similar behavior that is going to indicate they come from the same brand
of drones?
And to be really specific.
This is going to be a two sample t test, but we'll learn more about that later.
So the way this would really work is we would compare samples from the data distributions created from
both drones and use what's known as a P value to see the likelihood of being from the same brand.
So we'd take the data distributions from the real life experiments and we can actually begin to compare
them.
And as I mentioned later on in this section, we're going to dive into mathematics of how to conduct
such a test.
But I want you to also consider another situation or scenario that sampling and tests can help out with.
This particular situation was a situation where you can use sampling and experimentation to see, Hey,
do these two data sets come from the same population or what's the likelihood they come from the same
population?
Now in scenario two, imagine we're in a situation where we've tested the original drones, but now
I actually want to see if I can make modifications to them to improve their performance.
So it's technically still the same drone, but now I've affected it in some way.
For example, I add a really large propeller to the top of the drone, so technically the same drone,
but now a modification or something's been changed to it to have some sort of effect.
So we are now wanting to know if there are differences within the same population that is the same drone
brand after a change.
So this case is known as a paired t test, and this is essentially to explore the effects of the change
on the same population.
So we would run the experiments before and after the change and then perform what's known as a paired
t test, which is going to statistically tell us if we actually had an effect.
Understanding how to use sampling in conjunction with T tests, give us powerful tools to let us know
whether or not experiments worked and the ability to compare samples from data distributions, to compare
them against our assumptions like belonging to the same population, which in our examples was belonging
to the same actual brand of drone.
And remember, a lot of times with statistics and probability and experiments, a lot of these tests
only give you answers in likelihood, such as it's 99% likely that the two drones came from the same
brand.
You're never really going to get a 100% sure answer.
Now in the section, we're also going to be discussing what's known as the Central Limit Theorem, which
will reveal a fundamental property of data distribution and sampling from them.
Let's get a quick high level overview of the central limit theorem and why it's so important.
Formally defined.
The central limit theorem establishes that in many situations, when independent random variables are
summed up, their properly normalized sum tends toward a normal distribution, even if the original
variables themselves are not normally distributed.
So that's quite a lot to take in and it's a lot of jargon in there.
So I want you to get a clear understanding by actually showing the central limit theorem in practice.
So let's imagine that we have a data distribution that displays a bimodal behavior.
Bimodal essentially saying that by meaning to and modal for modalities is kind of a fancy way of saying
there's two humps on this particular data distribution.
You'll notice that one of them is centered around zero and one of them is centered around one.
Now, let's imagine the I were to take a random sample from this bimodal data distribution, and I take
that random sample.
So in this particular case, I grab a couple of points.
Obviously, probabilistically speaking, I'm more likely to grab points towards zero or one than those
in the middle where there's a low probability of picking them.
So after I picked those random sampling of points, I take the mean or average of that particular random
sample.
So in this case, let's imagine it's 0.43.
Then what I'm going to do is I take that mean value and I plot it onto its own plot and you can kind
of ignore the height of the line here.
What is really important is the x axis on that right hand plot.
So again, what we're doing here is I start off with this bimodal data distribution.
I take a random sample of points from it.
I take the average value of that sample and then I take that average value and I plot it onto its own
plot.
So all the ends up on the right hand plot is the value 0.43.
Then what I'm going to do is I'm going to keep repeating this process of taking that random sample,
calculating the mean of that sample, and then plotting it as a point.
So if I do this again, let's say this time I get 0.51.
And so I actually ran a simulation of this using the Python programming language.
And if we do this many more times, you will notice a specific behavior is going to arise.
And this is actually the central limit theorem.
So the central limit theorem states that the means of these samples is going to be normally distributed,
and it may be a little unclear using what is known as a rug plot, which is essentially just lines indicating
a specific point value.
That is the mean value of the samples.
So what I could do is create a histogram from the means of the samples rather than just doing this dashed
rug plot line plot.
So here I now have a histogram of the mean of those samples.
So again, to repeat myself, I took a sample from the data distribution on the left.
Then I calculated the mean value like 0.43 or 0.51, etc. Then I ended up creating a histogram of doing
that many, many times over.
And what the central limit theorem says is that histogram is going to be normally distributed.
Now, here's the cool part.
The mean of the samples will be normally distributed around the mean of the data distribution, and
that's actually why I chose this very specific bimodal data distribution.
You'll notice that if it's centered one modality around zero and the other modality around the one,
then the average of those values is going to be 0.5 right in the middle.
So that's zero plus one divided by two.
And if you keep doing these samplings and calculating of the means, you're going to get a normal distribution
of the mean of those samples centered around the mean of the data distribution, which is why upon simulating
this, you actually get a normal distribution that appears to be centered around 0.5.
So what is absolutely crucial about the central limit theorem or CLT, as you'll often see it shortened
to, is that this behavior is true for almost any data distribution.
One example exception is the couch distribution, but that's a very unique distribution that you probably
won't find yourself using in real life.
So while this is a very interesting property of data distributions and sampling them, the next obvious
question is besides being a neat mathematical trick, why would this property actually be useful in
the real world?
Well, in the real world, you often don't know the true distribution that your data comes from.
So think about this in terms of the entire population.
You often don't really know the true distribution of the entire population.
Maybe it's bimodal, maybe it's exponential, maybe it's normal, maybe it's has various modalities
like any of these examples that I'm showing here.
Well, with the central limit theorem, you can just say, well, I don't really care about that true
population distribution, because I know if I keep sampling from this data set, calculating the mean
and then plotting that, then I end up getting that normal distribution around the mean of the original
data distribution.
So the central limit theorem allows us to disregard the original data distribution since we know the
sample means will be normally distributed and we already know the normal distribution has a lot of unique
and useful properties for calculating statistics.
So if the normally distributed means, I can then actually perform the PT tests on the samples, which
is super useful.
So let's continue to learn more about the power of sampling and how we can use it to conduct tests.
We'll see you in the next lecture.