Everything will cover in this section and the next really depends on our ability to take a sample from
a population, which means that we want to start this section with a good understanding of sampling
sampling techniques and the bias that we can potentially introduce when we take a sample.
So we'll start by saying that a population is the entire group of subjects or information or people
or data that we're interested in, and we indicate population with a capital N, We've already talked
about population.
We've already seen instances where we refer to the population with a capital N this capital n value,
the value of N represents the number of people, subjects, members, data, etc. that are included
in the population.
And by definition a sample is a subset of the population.
So if this is our entire population, the entire universe of our population, then we want to understand
that a sample is just a subset of the population.
So maybe our population includes 5000 people.
We might be able to take a sample of, let's say, 700 people of those 5000.
And so the population size would be capital N equals 5000, The sample size would be lowercase, N equals
700, and those 700 people would just be taken from this larger population.
Now, of course, this whole idea of sampling exists because we're interested in studying the population,
but oftentimes it's very difficult or even impossible to actually collect data from every member of
the population.
We gave an example earlier in the course about studying the population of the state of California,
which is many millions of people.
If that's our population, it's going to be virtually impossible to collect data from the entire population
because as we're conducting our study or our survey, some people will move out of the state, some
people will move into the state.
New babies will be born, some people might pass away.
It would be essentially impossible to truly capture the entire population as a snapshot in one particular
instant and define that entire population.
And furthermore, the logistics of actually collecting responses from all of those people would be virtually
impossible, even if we could define who they were.
So for so many reasons, taking a smaller sample of the population of the state is going to make a lot
more sense.
Now, in a perfect world, we want this sample that we take to be a representative or unbiased sample.
And what that means is that, of course we want the sample to do a good job representing the population
and ideally we want it to be representative in every way.
So let's take a basic characteristic of the population, like gender, males versus females in the state.
Maybe the state is 54% female.
Well, then ideally we would want our sample to also be 54% female.
Maybe 20% of our population is between ages 35 and 50.
Well, we might want 20% of our sample to fall between ages 35 and 50.
We want the sample to represent the population because the idea here is that we would want to be able
to use the sample to make inferences about the population.
We want to be able to use what we find about the sample and scale it up so that we can make a proportional
conclusion about the population itself.
Let's take a really basic example and say that we have a big jar of marbles and the jar actually holds
20 measuring cups of marbles.
And we want to know the number of red marbles that exist in the jar.
And we want to do that without going through every single marble in the jar and counting all of the
red marbles.
So what we might do instead is take a one cup sample of the 20 cups in the jar.
And let's say that in that one cup sample we find that there are two red marbles.
If our sample is representative, what we're able to do is set up an equation like this.
Where we say for every one cup of marbles in the jar, we find two red marbles in that one cup.
And so in 20 cups, we should find how many red marbles.
Well, this is an equation.
We can solve it by recognizing that in order to change one into 20, we have to multiply it by 20,
Which means that in order to change, to add to this value, we also have to multiply it by 20.
So two times 20 is 40 and we say x equals 40.
In other words, 40 over 20 is the same thing as two over one.
And so we would make a conclusion that there might be 40 red marbles in the entire jar.
This is the kind of conclusion we might be able to make, assuming that this one cup sample of marbles
that we took from the 20 total cups in the jar is actually representative of those 20 total cups.
If this one cup sample we took is not representative, maybe there are no red marbles or there are many
more red marbles in this particular cup.
Then in another cup we might have chosen.
Then the sample isn't going to be representative and we're not going to get a good estimate of the total
red marbles in the entire jar.
So from here on, everything we talk about is going to be how to pick a representative sample and talking
about the bias that we might introduce when we choose a sample.
So let's start by talking about some different sampling techniques.
In our marble example, we had 20 cups of marbles in a jar and we just scooped out a single one cup
measurement and used that as a sample.
But in the real world, with real world kinds of problems, what are the different sampling techniques
that we can use?
Well, one of the most common, because it's the easiest, is called a simple random sample.
The advantage of using a simple random sample is that it can be an easy and therefore quick and cost
effective method of sampling.
And the idea is that we assign a random number to every member of our population.
And then ideally we use something like a random number generator to choose members from the population.
So for instance, if we're trying to study all of the employees in our company, let's say there are
1000 employees in our company, then we would assign all of the employees a number, starting with one
all the way through 1000.
So every employee has a number, and then we would use a random number generator.
We can find one online.
Excel also has a function that will randomly generate a number in Excel.
The function is this one here Rand.
Between as in a random number, between some certain interval and the inputs.
A and B are the smallest and largest numbers in our list.
So if our employees have been assigned a number one through 1000, then we would use the function equals
rand.
Between one comma, 1000 and Excel will randomly generate for us a number between one and 1000.
Using a random number generator like this one will pick for us random numbers, and that's how we will
build our simple random sample.
So the result might be employee number 17, 162, 414, etc. And we could keep going until we have whatever
sample size we need.
Maybe we want a sample size of 150 employees.
Of the 1000 total employees that we have, we would keep generating random numbers until we've picked
randomly 150 employees.
As we mentioned before, this sampling technique can be easy to implement sometimes and therefore cost
effective and efficient.
And it does a good job at truly picking random members from the population.
But there are a couple problems.
For instance, we do need to be able to assign a random number to every member of the population.
And if the population is so large or hard to quantify that we can't assign a random number to every
single population member, then it may be difficult to apply this method, and simple random sampling
might do a bad job ultimately creating a representative sample.
And that's because maybe our population of 1000 employees in our company is made up of exactly 500 men
and 500 women.
We would expect then that our sample might have about 75 men and 75 women, but it's completely possible
that using a totally random sample, we end up with a ratio like 130 men and only 20 women, which would
obviously not be a representative sample of the population, at least in terms of gender.
Now, a sampling technique we can use to potentially address some of the drawbacks of a sample.
Random sample is what's called stratified random sampling.
When we use this sampling technique, we break our population into strata.
So we set here for the sample random sample example that if we have 1000 employees in our company,
1000 total employees and 500 are men and 500 are women, instead of using a random number generator
to pick a sample random sample from all 1000.
In other words, instead of pulling our sample directly from this group, we could break the population
in half and have a population of men and a population of women, and then take a sample random sample
inside of the male population and a sample random sample inside of the female population to at least
control for gender.
So if our sample size is 150 people, we could take a random sample of 75 men and a random sample of
75 women, and that would give us a 150 person sample that is more representative of the company as
a whole in terms of gender.
If we're going to use this technique, we just need to make sure that our strata don't overlap.
So the strata need to be mutually exclusive.
For example, here we divided the company into 500 men and 500 women.
But let's say instead that we had divided the company by education level, classifying them by the college
degrees they held.
And we called one group bachelor's degrees and the other group master's degrees.
Well, if we have an employee who has both a bachelor's degree and a master's degree and we include
them in both strata, then we've tainted the exclusivity of these two groups.
They need to be mutually exclusive.
We just need to divide them along non overlapping lines and then we need to take a proportional sample.
So here we had two equally sized groups, but if our company instead had, let's say 750 employees whose
first language is English, and then 250 employees whose first language is something other than English,
and let's say we want to take a 100 employee sample, we would want to take 75 employees from the English
group and then 25 employees from the not English group to keep the samples proportional to the size
of each strata.
Then we would combine the 75 from this group and the 25 from this group to create our 100 employee sample
where we've controlled somewhat for primary language spoken.
Now, while stratified, random sampling can help to eliminate some of the sample selection bias that
we might create with simple random sampling by controlling for certain characteristics of the population,
this kind of sampling takes more work, and we might not have the resources or time or money or manpower
to perform this type of sampling.
And of course, just because we've controlled for one characteristic doesn't still mean that we're getting
a.
Perfectly representative sample.
For instance, we might control for gender, but maybe we're not controlling for household income or
language spoken or department of our company.
All of which might be important variables to consider when we're building our sample.
But these aren't our only options.
We can also take what's called a clustered random sample.
This sampling technique can be really useful depending on the situation.
For instance, maybe we're interested in learning something about the employees of Fortune 500 companies.
Well, it might be prohibitively difficult for us to get a complete list of every single employee at
all 500 of these extremely large companies.
So conducting a simple random sample might not make sense.
What we could do instead is say that we have 500 Fortune 500 companies, and we could think about each
of those companies as its own cluster.
So we have 500 clusters.
We could take a random sample of those 500 companies so we could assign each of the Fortune 500 companies
a number one through 500.
Then we could use a random number generator.
Let's say we want to take a sample of 25 of these companies so we would use our random number generator
to pick randomly 25 of the Fortune 500 companies.
So now we have 25 companies instead of 500 companies.
If we then survey every employee at those 25 companies and use them as a sample of all of the employees
at all 500 companies, we would call that a.
Single stage.
Cluster sample, but maybe this sample is still too large.
After all, these companies are enormous global companies.
So adding up all the employees at all 25 companies might still give us a huge list.
Instead, what we could do is take a random sample within each of these companies.
For instance, maybe we sample using a sample random sample, maybe we sample 100 employees from each
of these 25 companies.
If we do that, then we call this a double stage.
Cluster sample because we're sampling in two stages.
We started with 500 companies.
We consider each company its own cluster of employees, so we sample to choose randomly 25 clusters.
And then within each of those clusters, we pick out 100 employees in the second stage of sampling.
And if we have 100 employees at 25 companies, then the result is a 2500 employee sample, 100 times
25.
We get 2500 total employees and we might use this set of employees as a sample of all of the employees
at all 500 Fortune 500 companies.
But again, just like with the sampling techniques we talked about before, there are always pros and
cons here, right?
This can sometimes be a more labor intensive sampling technique.
And there's nothing really to say that these 25 companies are representative of these 500.
Maybe we would want to use a stratified random sample to pull these 25 from these 500 to help us control
for some characteristics.
There's also nothing to say that these 100 employees are representative of all of the employees at each
of these 25 companies.
There are always pros and cons.
The last sampling technique that we want to talk about is systematic random sampling.
This one's really similar to simple random sampling, and it's pretty easy to do.
We assign a number to every member of the population and then we just choose randomly.
We could use a random number generator for this, but choose randomly some starting number and then
pick some interval at which to pull members from the population.
So again, let's say we have our 1000 employee population at our company.
We assign each of them a number one through 1000, and then we use a random number generator to choose,
let's say the number 167, and then also the number 11.
Then to build our sample, we would start with employee number 167, and then we would just add 11 every
time to this previous number to get the next member of the sample.
So we would also then use employee number 178 and then employee number 189 and then employee number
200 211, etc. until we had the total number of employees that we wanted to include in our sample.
So we've introduced these different sampling techniques.
But remember here that practically speaking, our goal is never really to pick a sample that's absolutely
perfectly representative in every way of our population.
That's almost certainly going to be impossible.
We can't control perfectly for every characteristic of the population.
Instead, our goal is always to think through what we're trying to study, what characteristics we have
in the population, what's maybe most important to try to control for, And we're trying to build the
most unbiased representative sample that we possibly can.
And then and this is critical pair that with being completely transparent about our sampling technique.
So we do our best to accomplish this goal of finding a representative, unbiased sample that will let
us make good inferences about the population.
But whatever sampling technique or set of sampling techniques we use to get to that sample, we just
overly communicate about exactly what we did, how we built our sample, what methods we use, etc.
so that anyone who is trying to interpret the findings of our study can get the clearest possible view
of the limitations that are attached to our results.
In fact, we ourselves might even want to offer up anything that we can spot about the limitations of
our sampling.
For instance, if we took a clustered random sample like this with the Fortune 500 companies, we might
want to point out right up front that we used a random number generator to pick 25 of the Fortune 500
companies, but that these 25 might not be representative of the total 500.
Maybe.
We look at the 25 we picked and they are disproportionately concentrated in one or two geographic regions.
And so we might want to point out, along with our result, our conclusions, that we still have questions
about the geographic representation of our sample and that the lack of accurate geographic representation
may be skewing our results.
So we're trying to be as representative as we can be paired with as much transparency as possible about
our sampling techniques.
Now, all that being said about sampling techniques, let's talk about some of the bias we might introduce
depending on the sampling techniques that we use and how we engage with our sample participants.
So when it comes to actually building our sample, we want to be careful about voluntary response or
self selection bias.
And there are so many different types.
Biases we can introduce.
We want to cover just some of the most obvious ones here, but just know that this is certainly not
a complete list.
So the idea with voluntary response bias or self-selection bias is that if people are asked to voluntarily
participate in a study or a survey or they self-select for participating, we ask people to participate
and some people voluntarily participate and others don't.
By definition, we're getting people who are more willing to participate, and that could skew our results.
For instance, maybe we want to ask students about their study habits and we allow students to voluntarily
participate.
Well, maybe students who already have better study habits, who are better students, are more eager
to participate in a study like this one than students with worse study habits or students who maybe
aren't doing as well in their classes.
Which means we're going to end up with a sample of students who maybe study more, are more successful
or getting better grades.
That's going to affect the results of our sample and maybe make our sample not representative of the
entire population.
We can also think about non-response bias, which is all about people who choose not to respond to the
survey or to a study.
So if a political representative asks his constituents to fill out a survey and mail it back, he might
get fewer responses from busier people, people who are working parents, caretakers, people who travel
for work and aren't home as much and don't get their mail, don't have time to fill out the survey and
send it back.
Those groups may fail to respond to the survey and our sample is going to be biased.
Similarly to non-response bias would be this idea of under coverage where we just don't collect data
from entire groups of the population that actually should be included in our study.
Maybe we run a daycare center and we're surveying parents as they come to pick up their children, but
maybe whoever is conducting the study only stays at the daycare center until 5:00 PM.
Well, any parents who pick up their children after 5 p.m. won't be represented in the study.
They were just completely missed as a group.
And maybe it's the people who pick up their children later who make more money or less money.
Maybe they're more likely to work full time or more likely to work part time.
Maybe they're more likely to be single income households or dual income households.
We could be missing a whole group of our population.
And so we have this under coverage risk that's going to bias our sample.
We also have to worry about prescreening or advertising bias.
How do we get people to participate in our sample in the first place?
Going back to the idea of surveying employees at our company, if we post a flyer in the cafeteria of
our company offices, and that's the only place that we advertise the survey, then the only people
that we're going to get included in our sample are people who saw that flyer, which means people who
use the cafeteria, who work when the cafeteria is open and choose to visit the cafeteria, which means
there are people who maybe didn't pack their own lunch and bring it to work.
We're setting ourselves up for a certain kind of sample.
We're introducing bias just in the pre screening process or in the way that we advertised for this study.
And then the last one that we'll talk about.
But remember, these are certainly not all of the different types of bias we can have is survivorship
bias.
Maybe we're trying to understand more about what the business environment looks like today in present
time, but maybe what we miss is that the business environment, the business climate today is very
difficult.
It's a tough economy and maybe many businesses have just gone out of business and we don't survey or
we don't include any representation of those businesses who just closed, who just shut down.
They were fully operational last month, but in this difficult environment they've closed.
And so we're not including them in our sample.
And we're introducing this idea of survivorship bias.
This could also kind of be an example of under coverage where we're not covering that group of businesses
that just closed.
Also potentially similar to non-response bias, and many of them are related to this idea of convenience
sampling where we sample in a way that is convenient and easy for us but actually doesn't build a very
good representative sample.
We want to avoid choosing a sample based on convenience and instead start with an idea about what a
good representative sample might be.
We want to think about the best way to collect a good sample, not how to sample in a way that is most
convenient, most easy, fastest, cheapest for us.
Because if we just sample in a way that is convenient, we may not get a representative sample at all.
So these are all things that we need to be worried about when we are building our sample.
But once we have our sample and now we're engaging with the participants in our sample, we can introduce
all.
Kinds of problems into the data, just by the way that we engage with the members of our sample.
For instance, we can introduce what's called measurement bias, where maybe there's something wrong
with the tool that we're using to measure data or collect data from our sample.
For instance, maybe the data that we're collecting is the amount of time that our employees work,
and we don't realize that when they're clocking in for work and clocking out of work, the clock that's
keeping track of that time is actually slow.
And so it appears as though they're working for longer than they actually are.
The tool that we're using to measure the data that we're collecting is wrong.
It's inaccurate.
It's not well calibrated.
That could throw off all of our data.
Even if we have a great representative sample.
We can also introduce something called social desirability bias.
If we are asking questions of the people in our sample, we have to realize that people have this social
desirability bias.
They want to be seen in a favorable light.
So if we ask them difficult questions, for example, have they ever stolen something?
Or when was the last time they lied?
They may not be truthful.
They may not give us an accurate answer because they're tempted to answer the question in a way that
makes them look better.
So we have to be aware that the data that we're collecting around that question could be skewed toward
a more favorable response.
Somewhat related, we can have this idea of leading questions where the way that we are asking the question
of our participants leads them to answer in a particular way.
Maybe we're asking people about how they'll vote in the next election, and instead of asking them who
they plan to vote for, we ask them whether they'll vote for the more experienced candidate.
Of course, we think about the more experienced candidate being the better candidate.
And so if we ask the question that way, people might tend to sort of agree with us that they're going
to vote for the more experienced candidate when they actually don't know who that is or they're not
planning to vote for that candidate because it doesn't match their political party.
We might be better off just asking them who they're going to vote for rather than leading them or prompting
them with this idea of will you vote for the more experienced candidate?
And then we can have something like recall bias where if we're asking people to remember something or
give us an account of something that happened in the past, we know that human memory is actually notoriously
bad.
People will recall events in one way when the past actually unfolded differently than their memory.
So asking people about the past is predictably risky.
We introduce this idea of recall bias now, just like these biases we talked about for building the
sample.
This is certainly not an exhaustive list of biases that we can introduce when we're actually surveying
the members of our sample.
But it gives you an idea of some of the things that we want to be aware of.
And to summarize, like the idea that we talked about before, the goal here is not to perfectly eliminate
every kind of possible bias that really is impossible.
Instead, we want to do everything we can to introduce as little bias as possible and then at the same
time be as transparent as we can about how we built our sample and how we collected data from that sample
so that everybody looking at and interpreting the data can have as much information as possible about
how much we can rely on the data from the sample actually representing the population.