Covariance and correlation both inform us about the relationship between two data series, and while
correlation is the value we end up being more interested in because it's a more helpful metric, we
have to calculate covariance on our way to correlation, which is why we want to start with an understanding
of covariance.
So the covariance formula will start here and then unpack it piece by piece.
But the covariance formula, this says that the covariance between two series given by X and Y is equal
to and this right hand side basically just says the mean of the square areas that are given between
each particular data point x sub by sub I and the mean data point x bar y bar.
Equivalently, we can write this formula this way.
So these two formulas represent exactly the same thing.
They're just written in different ways.
This second formula here tells us that covariance can also be calculated as the mean product.
So if we multiply each x sub by by its corresponding y sub by and then we take the mean of those products,
we get this value here and then we subtract from that the product of the means.
So the mean of all of the x sub values, the mean of all the Y sub values, and we multiply those two
products together to get the product of the means.
We subtract that from the mean of the products.
This is another way to calculate covariance.
So when you see these two different formulas, they mean the same thing.
Now, to understand covariance, what we really want to realize is that this first formula looks a lot
like the variance formula that we learned about earlier To get back from this formula to the variance
formula.
What we want to do is replace Y with X.
So we would say the covariance of x with itself one over n times the sum of i equals one to n of x sub
i minus the mean times x sub i minus the mean.
Because again we're just replacing y with x here and then we can see that we can rewrite.
This right hand side as the square of this single binomial because we have two identical binomial is
multiplied together here.
So instead of saying x abi minus x bar times the same thing, x sub minus x bar, we can rewrite that
as x abi minus x bar quantity squared.
And we should now recognize this as the variance formula for the variance of x.
And so because these right hand sides are equal, what we're essentially saying here is that the variance
of a single variable X is equivalent to the covariance of that variable with itself.
Given that relationship, if we sort of work backwards from this idea of variance.
So thinking about variance, if we almost just think about a one dimensional number line and we have
some data set, some data series given by this X variable and we say that the mean of that series is
here at X Bar when we want to calculate the variance of this variable X, what we do is we look at all
of the other values of X in the data set.
So let's say that there's another value of X in the data set right here and right here and right here.
And what the variance formula calculates is the mean of the square areas that are generated.
When we make a square out of the distance between each of these data points and the mean.
So in other words, looking at this closest data point here, if we create a square between this data
point and the mean and then we create another square between this data point and the mean and finally
another square between this data point and the mean.
Now we have three pieces of area.
This Green Square is the largest amount of area.
Then the white square and then the blue square is a smaller amount of area.
What this variance formula gives us is the mean amount of area of these three squares or the average
area.
So let's say that this little blue square, let's just make up values.
Let's say that this little blue square has a square area of one square unit.
Let's say that this white square is a square area of four square units.
And the Green Square is a square area of five square units.
What we would do is add up all of these areas.
So five plus four is nine, plus one is ten.
The total area of all three squares together is ten square units.
And then we have three data points.
So we would divide that by three.
We get about 3.33.
And so the variance of the x variable here, if this is the data set, is about 3.33.
This is what we calculated in the past, the variance of just one variable x, which we now know is
also equivalent to the covariance of X with itself.
But instead of calculating the covariance of X with itself, if we instead want to calculate the covariance
of X with a different variable y, now our little two dimensional picture has to transition to a three
dimensional picture.
So in that three dimensional picture, instead of plotting values of just X along this one dimensional
line, we would plot x, y, coordinate points in a two dimensional plane.
So instead of just now along a line where we have this mean value of X here and then we have this point
x sub one, x sub two and x sub three, now we have coordinate points where maybe this first point here
is the point x one, y one and this point up here is the point x seven.
Why seven But each of these is just one value from the X Data series paired together with its associated
y value from the Y data series and then plotted in the two dimensional plane.
Now what we already know for this series, instead of having just a mean X bar for the one series in
X, when we find a mean for X and Y together, what we're doing is we're averaging all the X values
and we're averaging all the Y values and then we are plotting that mean point.
And this point is X bar.
Why bar It's the mean of all the X values and the mean of all the Y values.
And we plot that point and we can think about this as the balancing point of the entire data set.
Then just like when we had variance in one variable to show covariance with two variables, we just
look at the sum of all those square areas.
So if we plot some of those square areas, we connect each individual point to that mean point.
So we did that here with this point.
Here with this point, we found this smaller square area.
With this point up here, we found a smaller square area.
And then with this last point up here, one of the bigger square areas.
So we plot all those squares.
If we add up all of the areas of all the squares connecting each point to the mean, then we get this
whole sum right here.
When we divide by n the number of data points that gives us covariance, which is just the average of
all the square areas.
So this covariance concept is just like the variance value we learned about earlier, except that in
this way we get to see how X and Y varied together.
We're not just looking at one variable on its own.
Instead, here we're looking at the two variables in comparison to each other.
And we can sort of imagine here how both X and Y have an effect on the variance.
This is our mean point, but if we picked a point, let's say way up here, way above the mean, but
not too far to the right of the mean.
And then we created this rectangular area between that point and the mean point.
Maybe that looks something like this.
We can see how this area might actually be quite large because of the amount of variance in Y between
this particular point and the mean.
Even though the variance in X for this particular point is small, the fact that the variance in Y is
larger lets us see that this point actually varies quite a bit from the mean, even though the variance
in x is small.
If we were just looking at one variable alone, just the variance in x this point wouldn't look like
it varied that much from the mean at all because it's only this tiny little width right here that constitutes
the variance in x.
But this huge height gives us the variance in Y.
And so we can see that this point actually varies quite a bit from the mean.
And so it's that covariance being able to see what's happening to X and Y at the same time.
That allows us to see more than just this one variable picture over here.
Now, when we calculate a value for covariance, it will help us to see the relationship between the
two variables X and Y.
For instance, with this particular data series here, we can actually sketch a line through the data
and we'll talk more about this later, but sketch a line through the data that shows us that there's
a positive relationship between X and Y.
It's positive because as X increases.
So as we move to the right along the horizontal axis, as X increases Y also increases, we move up
along the vertical axis.
Or we could work backwards and say that as X decreases.
So as we move from right to left this way, Y also decreases.
We move from top to bottom.
So either Y is decreasing as X is decreasing or Y is increasing as X is increasing.
In either case, both variables are moving in the same direction at the same time.
That's what we call a direct relationship or a positive relationship.
If we have instead a relationship moving in the other direction.
So if our line looked something like this instead, maybe like this, that would indicate an inverse
relationship or a negative relationship, because as X is increasing.
So as we move from left to right and the value of X increases, the value of Y decreases, we're moving
down along this line as X increases or working in the other direction as X decreases.
So as we move from the right to the left, as X decreases, Y is increasing.
And so the variables are moving in opposite directions and we call that an inverse or a negative relationship.
The value of covariance will give us that general direction of the relationship between the two variables.
If covariance is positive, that means there's a positive direct relationship between X and Y.
If covariance is negative, that means there's a negative or inverse relationship between X and Y.
So if covariance is positive, the variables generally are moving in the same direction.
If covariance is negative, the variables in general are moving in opposite directions.
For that reason, covariance is a helpful metric.
The problem is that it doesn't tell us the strength of the relationship, only the direction of the
relationship, and it's not a standardized value.
So changing the scale of our data will change the value of covariance.
One of the ways that we can see that is by simply adding one data point to this existing data set.
So we had all of.
These little blue points in our data set, and they're all on or very close to this line running through
the data set.
And we would calculate a covariance by finding the mean of all of these different square or rectangular
areas.
But if we add another point to our data set and let's say that it is directly along this line right
here, even though this point is right in line with each of these other data points in the set, this
point, this point, this point here, here and here, we're all right along this line.
And so is this new point such that this new point isn't adding really any new information than all of
these other points here?
It's right in line with all of the other values along this line.
But if we sketch in the area created between this point and the mean right here, we can see that this
massive amount of area is definitely going to increase our covariance calculation, even though this
particular point doesn't really vary off of this trend line in either the X or Y direction.
And we could continue adding points directly along the line way out here in space.
And that huge area is going to contribute to increasing our covariance value, even though we're not
really seeing any variation in the set.
Essentially all we're doing is scaling up this sum right here, which means that we calculate a larger
total area and that tells us that our data points are creating a large amount of area away from the
mean point.
But it doesn't give us any indication about how close or far our data points are from this trend line.
And that's what we're really interested in.
We want to know if all these data points that we're adding to the set are right along this trend line
or far from it.
So at this point we added up here was right along the trend line.
We want to be able to distinguish between that kind of a point and maybe a point down here that is far
away from the trend line.
And that's what correlation is going to allow us to do later.
But for now, despite all of its problems, let's just look at an example so that we know how to use
this formula to calculate covariance.
Now that we know what it's doing, we can obviously do this by hand or with a calculator.
But these kind of calculations, especially as our data sets get larger and larger, are much, much
easier to do with software.
So for instance, here's a table, and these are the values in our X and Y series.
So we have our X values here one, two, three, four, five, three and four, and their corresponding
Y values two, four, six, eight, ten, five and nine.
So essentially these are all of our x, sub I and Y sub AI values.
And then we need to find the mean x bar and y bar.
So we sum the entire X series and the entire Y series and we get sums of 22 and 44 respectively.
And then because there are seven data points, because PN is seven, in this case we divide 22 and 44
by seven to get the mean.
So this here is X bar and this is why bar.
And in fact these series are what was represented here in this original data set of blue points in the
plane.
So we calculate the mean of both the X series and the Y series.
Then we need to calculate these x sub minus x bar values and the Y sub minus y bar values.
So that's where we go next.
We have x sub, B minus x bar.
So this first value here is one minus the mean of x 3.14, etc. This second value here is two minus
the mean, this is three, minus the mean etc..
And then same thing here for y we take y of two minus the mean of 6.28 to get this -4.28 value.
In this second row we take four minus the mean, we get a -2.28.
So we calculate all those y minus y bar values.
So we calculate all those y sub by minus y bar values.
And then of course we take their product because here we have to multiply these x sub minus x bar values
by the Y sub by minus y bar values.
So once we have these products, we then add them all together.
This sum here says to add all those products together.
That's this sum right here.
So when we add everything together, we get 22.7 approximately, and then we just divide this value
by seven to get the covariance of X and Y as approximately 3.2449.
This is a rounded value here, but this is the covariance of x with Y.
And like we said at the beginning, this second formula here will do the same thing.
All we do once we have our x and y value.
I use.
We take the product of each X and Y.
So here we have our values for X, our values for Y.
So if we multiply one times two, we get two.
Two times four is eight, three times six is 18, four times eight is 32, etc. All the way down to
four times nine is 36.
So if we multiply all the X and Y values, we find all these products.
So this here is the x y column and then we take the mean of these.
So we add all of these together and then we divide by an equal seven the number of data points we have.
So if we add all these together and then we divide by seven, the result we get is this mean of the
Z products, which turns out to be 23, and then we just subtract from that the product of the two means,
which we already have, we already have X bar and Y bar.
You can see here we have about 3.14 and 6.28.
If we multiply these values together, we're going to get about 19 and three quarters.
And so when we then take 23 here.
Minus the product of X bar and Y bar, which is about 19.75.
Roughly.
The value we end up with is about 3.25 or more specifically, this exact value here for covariance 3.2449
if we rounded the first four decimal places.
So these two covariance formulas do calculate exactly the same value just in two different ways.
And so just to tie this covariance back to this data set, we started with the fact that we get a value
here of 3.2.
The fact that it's positive tells us that there is a positive or direct relationship between X and Y,
meaning that as X increases, Y increases or as X decreases, Y decreases.
And we see that with this rough trend line that we sketch through the data.
If we had found a negative value for covariance, it would indicate the opposite relationship between
the X series and the Y series.
It would tell us that there was more area over here in this direction.
Down here or.
Here along this negative trend line.
Then there was area in this direction along the positive trend line.
So really the biggest takeaway here is that when we find a positive covariance value, it means we have
a positive or a direct relationship that the variables change in the same direction instead of in opposite
directions.
And next, we're going to look at how to use what we've learned about covariance and standardize it
into correlation, which will be an even more useful measure for us.