Earlier in the course in the joint distribution section, we talked about covariance and Pearson's correlation
coefficient.
Now we want to revisit that idea of the correlation coefficient to talk about that and the residual
which are both key parts of regression.
The focus of this section.
So if you remember Pearson's correlation coefficient, which we often indicate with this variable R
is equal to one over n minus one.
Where n is that sample size multiplied by the sum of these two products here x sub minus the mean of
x divided by the sample standard deviation of x and then y sub by minus the mean of y divided by sample
standard deviation.
With respect to y, realize here that we could also write this formula this way because both of these
values here are basically the Z scores for X and Y.
If we look at these here, we recognize them as the formula for a Z score X minus the mean of X divided
by its standard deviation is the Z score for X.
So we can replace each of these fractions with the z score of x, the z score of Y.
So this is another way to write the correlation coefficient.
And then this formula here is the one that we used earlier when we talked about the correlation coefficient
and covariance.
All three of these get us to the same place so we can use whichever formula is most convenient.
Remember that the value of the correlation coefficient always falls on the interval -1 to 1, so we'll
never find a value for the correlation coefficient that is less than negative one or greater than positive
one.
The correlation coefficient will always take on a value somewhere in the interval, negative one to
positive one.
And remember that the value of this correlation coefficient just tells us how strong the relationship
is between these two values x and y.
So if we think about a scale for the correlation coefficient, so we could almost sketch a number line
like this where the correlation coefficient is negative one over here, positive one over here and zero.
At this point the closer we are to negative one for the correlation coefficient, the more we say that
a regression line with a negative slope perfectly describes the data.
So if the correlation coefficient is exactly negative one, it means that all of the points in the scatterplot,
all the points in the data set lie along the regression line.
Exactly.
They all lie directly on the line.
None of the points are away from the line.
So for instance, if we say that this is the regression line, maybe it looks something like this,
then all of the points in the scatterplot lie directly on this line.
There are no points away from the line at all.
Whereas if the correlation coefficient is equal to positive one, that means that a regression line
with a positive slope perfectly describes the relationship in the data, perfectly describes the position
of all the points in the scatterplot.
So that means that all of the points in our scatterplot lie exactly on this regression line.
If the correlation coefficient is exactly zero, that means that a line doesn't do a good job at all
of describing the relationship in the data, which means our scatterplot might look something like this,
kind of like a blob.
There's no obvious linear relationship here at all.
We can't really identify any line that starts to describe some kind of trend in this scatterplot and
if we lie somewhere in between.
So if R is between zero and negative one, that means that our regression line has a negative slope.
So it might look something like this, but that it doesn't perfectly describe the data.
So we might have some points along the line, but we'll also have some points away from the line like
this.
But we see this general negative trend.
And then of course that means that a correlation coefficient between zero and positive one.
So somewhere over here means that the regression line is going to have a positive slope.
So maybe something like that, but that the data will not lie all along the line.
So we might have some points on the line, but we'll also have some points away from the line like this.
Don't confuse the value of the correlation coefficient r with the slope of the regression line.
So imagine here all of these regression lines, this regression line, this one we already sketched
and this one right here.
If we had scatter plots for each of these regression lines where all the points in the scatterplot.
Perfectly along the regression line, then each of these scatter plots would have a correlation coefficient
of one.
But you can see that the slope of the regression line is different for each plot.
In other words, a correlation coefficient between zero and one just means that the regression line
has a positive slope and a correlation coefficient between zero and negative.
One means that the regression line has a negative slope, but the value of the correlation coefficient
doesn't actually tell us the slope of the regression line.
All it tells us is that the closer we are to positive one, the more tightly clustered the points are
around a regression line with a positive slope, and the closer the correlation coefficient is to negative
one, the more tightly clustered the points in the scatterplot are along a regression line with a negative
slope.
Now remember that to calculate the correlation coefficient, we can use any of these formulas here.
If we use this third one, we can create a small chart to quickly make this calculation for us.
And here's what that chart should look like.
So if we have raw data given by these columns X and Y, so these are the columns with our raw data,
the values of X are zero two, four, six, eight, ten and 12, and the values of Y that correspond
to each of those X values are given in this column here.
In other words, we have coordinate points here x, y.
So one coordinate point in the scatterplot is 00.8.
Another coordinate point is to one, another coordinate point is four, 0.2, etc. So we have all these
coordinate points.
You can see that there are seven data points.
And so if we're using this formula to find the correlation coefficient, you can see here that we're
going to need the mean of X and the mean of Y, which means that we sum all the values of X, we get
42 and then we divide by the number of data points we have, which is seven.
We have seven data points here.
So we divide 42 by seven and we get a mean of six.
We sum all the values for y divide by seven and we get a mean of 0.8.
So that means that the mean of x here is six, the mean of Y is 0.8.
So we have those two values and then for each value of x, we need to find x sub B minus the mean and
for each value of why we need to find y sub B minus the mean.
So these next two columns here to get this negative six, we take this particular value of x zero and
we subtract the mean so zero minus six to get negative four, we take two minus six.
To get negative two, we take four minus six, etc..
So to find each of these values here, we subtract the corresponding mean from each data point and we
get these two columns.
Now our formula tells us that we have to multiply each of those values together, so each x value multiplied
by each y value.
So negative six times zero gives us six, negative four times 0.2 gives us -0.8.
So we find that product.
In other words, when we multiply these two together, we get this product here and then summing all
of those products.
So to find this sum or the entire numerator here, we take the sum of all of these products and we see
that sum right there.
So 1.6 is the value of our numerator.
And then in the denominator we again have these x sub minus the mean of x values, but we need to square
them.
So we use this column here to square all the values in this column.
In other words, we are getting this squared value here in this column and we're getting this squared
value here in this column, and then we need those sums.
So to get this sum here and then separately this sum here, we need to add all the values in each column.
So we get that sum and that sum.
And then taking the square root of the product of those two is the same as taking the square root of
each of them individually and then multiplying them together so we can take the square root of both
of these values.
So we get those square roots here and here.
And if we multiply these together, we get the denominator.
So this entire denominator here is the product of these two values here.
So if we multiply these two together and then we take 1.6 divided by the product of these two, we get
the correlation coefficient.
R And so the correlation coefficient for this data set here of x and Y values this raw data, our correlation
coefficient is 0.101.
So the fact that this value is positive tells us that the regression line through this scatterplot is
going to have a positive slope.
But the fact that this value is very close to zero instead of positive one.
So the value is maybe right about here means.
The correlation is very weak.
We don't have a strong correlation like we do over here where all the points in the scatterplot lie
exactly on the regression line.
Instead, the points are very spread out away from the regression line, but the regression line does
have a positive slope.
So if we replace this table with a scatterplot of the raw data, then we can see the relationship we
were talking about.
Here's the regression line through the scatter plot of that raw data that we were just looking at.
We can see that the regression line does have a positive slope because as we move to the right, the
line moves up.
So we've got a positive slope, but the data is very spread out away from the regression line.
It's not tightly packed or tightly clustered close to the regression line.
So it does make sense that we would find a correlation coefficient that is very close to zero, but
still positive to indicate the positive slope of the regression line.
That being said, as a general rule of thumb, we do classify the strength of the correlation based
on the value of this correlation coefficient.
So in particular, if the correlation coefficient is specifically between negative one and -0.07, we
tend to call that a strong negative correlation.
Whereas if we're kind of in that mid negative range between negative point seven and -0.3, we would
call that a moderate negative correlation.
And then if the correlation coefficient is between -0.3 and zero, we would call that a weak negative
correlation.
And then we have the mirror image of those relationships on the positive side.
So between zero and 0.3, we have a weak positive correlation between 0.3 and point seven, a moderate
positive correlation and between 0.7 and one a strong positive correlation.
So our correlation coefficient of 0.101 would fall within this range right here.
And so we would say that this data has a weak positive correlation.
Now related to the correlation coefficient.
At this point we need to talk about the residual as well.
So the residual for any data point, In other words, every data point has a residual.
The residual for any data point in a scatterplot is the difference between the actual value.
So the data point itself and what the regression line predicts at that same value of the independent
variable.
So if we say here in this scatterplot that this is the horizontal x axis, this is the vertical y axis
here we could look at as an example this data point here, its actual value is two, but it's predicted
value is what we find along the regression line.
If we follow this data point down to its corresponding point along the regression line.
So it looks like the predicted value.
So this is the.
Actual value.
And then this value right here is the.
Predicted value.
It looks like this predicted value is maybe about 0.85, something like that, roughly.
So the residual is the difference between that actual value and the predicted value.
We can write it this way.
The residual is actual minus predicted or mathematically we would write that this way because we often
also think about the residual as error.
And so we represent the residual with the variable E And then the actual value here is the actual y
value of the data point.
Remember, this data point here is a coordinate point x, y, and so it has its actual y value, but
then the corresponding y value along the regression line.
Remember that we write the regression line or the equation of the regression line.
As for y hat, the equation of the regression line is y hat equals m x plus B, where m is the slope
of the regression line and B is the y intercept.
So the predicted value along the regression line is always given by y hat.
So if we look at the residuals for all the points in our scatterplot, we would show them like this.
These are all the residuals.
It's this difference.
We could also think about it as distance, the distance between each point and the regression line.
So the points that are closer to the regression line will have a smaller residual.
The points that are further from the regression line will have a larger residual.
If the point in the scatterplot is below the regression line, its residual will be negative.
So everything down here, all four of these points here will have a.
Negative residual, whereas these three points up here will have a positive residual because of their
position in relation to the regression line.
And now that we understand this idea of the residual, we can define the regression line in a new way.
What we can say is that for any regression line, any y hat equation representing a regression line,
we know that the sum of all the residuals is equal to zero.
That is another way to define or identify the regression line.
The regression line will always be the line that makes the sum of all of these distances equal to zero.
That also means that the mean.
Of the residuals is equal to zero.
And in general, that should make sense because the idea of the regression line very, very broadly
is that it's trying to balance out all of the data points in the scatterplot.
And if we're creating balance in the scatterplot, we can think about that as trying to balance all
of these residual values, the distance of each point to the regression line.
And so the mean of the residuals is going to be zero.
The sum of the residuals is going to be zero.
Now, we've already looked at in the past how to calculate the equation of the regression line.
The equation of this regression line in particular is y hat equal to 0.0 143x plus 0.7143.
Both of these values for M and B are rounded to four decimal places, but this is the regression line
equation.
And so what we can do once we have the regression line equation is create a table of actual and predicted
values and then use those to calculate the residual for each data point so we can create a chart.
These first two columns are all of the raw data that we looked at earlier.
These are all of the X values and then these are all of the Y values or the actual values for our equation
here.
The predicted values are the values we get from our y hat equation, our regression line equation.
What we do is we take each value of x and we plug it into the y hat equation.
So you can see here, if we plug in x equals zero, then this term goes away and we're left with y hat
is equal to 0.7143.
And we see that here 0.7143.
So we're getting these predicted values out of the y hat equation and then we know that the residual
we can calculate as Y minus.
Y hat.
And so we find the value of the residual.
We find the value of the area.
And we see here, in fact, these three positive values that represent the three points above the regression
line and the four negative values that represent the four points below the regression line.
Now, the reason that we're looking at this chart is because we can use it to illustrate these two facts
here that we just talked about, that the sum of the residuals is zero and that the mean of the residuals
is zero.
Because if we create a new scatterplot where we have our X values along the horizontal axis, just like
up here, but now along the vertical axis, we plot the value of the residual.
So notice here that in our vertical axis, the value of zero is here in the middle.
And so we have some negative values below zero and some positive values above zero.
If we plot each of these values of the residual in this new set of coordinate axes, we get this scatterplot
here.
Ignore the yellow line for a second.
We get this scatterplot with all these new points.
What we see is that they all correspond in terms of their position to the scatterplot up here.
We've just changed the vertical axis.
So now this point, instead of being plotted with a vertical value of two, is being plotted with a
vertical value of about 1.17.
We see that point here.
These two points down here are the -0.57 and the -0.60.
In other words, up here in this scatterplot, all of these points are plotted as x, y coordinate pairs
down here in this scatterplot, all of these points are plotted as x, y hat coordinate pairs.
And when we use this new set of coordinate points, when our coordinate points are given by X and Y
hat and we use that set of coordinate points to find a new regression line equation, that new regression
line equation is this line here, we'll call it y hat of the residuals.
That new regression line equation is the equation y hat equals zero.
In other words, another way to put this is that if in this plot, the points in the scatterplot are
pretty well spaced out with a bunch above, this y equals zero line and a bunch below this y equals
zero line.
That means that a linear model is going to do an okay job describing the trend in the original scatterplot
with the original set of data.
So if you remember, we said before that this regression line, this line of best fit, this line of
least squares, we have a bunch of different names for this regression line, this trend line.
We said before that we were always trying to minimize the sum of squares so we would look at each data
point in the scatterplot and we would think about creating a square between that point and the regression
line.
So we would have a square like this and then we would have a square like this, and then this one up
here would be really, really big.
This square here would look something like that.
Let's just go ahead and make this big one.
So we have this big square.
We said before that we were trying to minimize the sum of all of these square areas.
Well, now that we know that the side of each square can be described as the residual E, if we just
say that e sub n represents all of the residuals and we square that value, this is all of the squared
residuals and this is the sum of all of the squared residuals.
This is the value we're trying to minimize to find the very best fitting line through the data, through
the scatterplot, to find the equation of the regression line, to find the equation of the trend line,
the line of best fit, the line of least squares.
This is the value we're trying to minimize when we describe that value in terms of this new idea we've
introduced, which is the idea of the residual Thinking about this in the context of the residual also
gives us a better idea of why we look at the squares instead of just the pure distances.
In other words, instead of just pure actual minus predicted the distance from the data point to the
regression line.
We instead look at the squared area, and the reason is because in the context of the residual here
we get a positive residual for any points that are above the least squares line.
We get a negative residual for any points that are below the least squares line.
And if we have a bunch of points above and below the line, those positive and negative distances would
cancel each other out if we didn't square the values to turn them into areas.
So if for this point we got a residual of positive one, and if for this point we got a residual of
negative one, those two things would cancel each other out completely and we wouldn't necessarily be
optimizing for either of those data points.
When we square the residuals, we get two pieces of squared area.
They both count as positive pieces of area.
And so we can make sure to account for minimizing both of those areas when we're building the equation
of our regression line.
So we're always trying to minimize the sum of the squared residuals or the sum of these square areas.
So that process of trying to minimize the residuals by minimizing the squares of the residuals, that's
where we get that name least squares line or line of least squares, or we often call this process least
squares regression.
We're just trying to minimize the total area of all of these squares.