Welcome everyone to this section of the course discussing joint distributions, more specifically,
covariance and correlation.
When studying mathematics for data science, we're often interested in the relationships between two
data dimensions or data features.
Data driven organizations often need to understand interactions between two data features in order to
make decisions about the features that are linked.
Joint distributions are going to allow us to mathematically quantify the relationship between two distributions
of data.
So here we can see a visualization of the relationship between total bill that somebody leaves after
a meal and the tip that they leave.
And we can see that they seem to be linked somehow.
So there's an intuition already that is total bill increases, so does the tip.
So this section of the course is going to focus on two specific properties of joint distributions,
covariance and correlation.
In general, these terms measure how much two data features move together.
For example, the total bill moving along with tip and likewise the tip moving along the total bill.
So it's a common task to try to understand correlations between two data variables.
For example, most businesses are going to keep track of how correlated price changes are with overall
sales revenue.
When thinking about covariance and correlation, we're typically discussing it in terms of two data
features.
Now, I want to point something out about the notation, especially if we're thinking about something
like machine learning when we're talking about covariance and correlation and the formulas behind them.
We commonly notate the two data features as X and Y.
Since if one were to plot these features, you could plot one on the x axis and the other on the y axis.
So, for example, if we look back at our TIPS data set here, we can see Total Bill is a feature and
we're plotting it along the x axis.
And Tip is also a feature that we can plot along the Y axis.
So as we continue to actually show you the formulas for covariance and correlation, we can think of
total bill as X and tip as Y.
Now, before we do a deep dive into the details of covariance and correlation in this lecture, we want
to do a high level overview of these concepts.
We'll also understand the motivation behind understanding these concepts in the context of using data
science for business.
So let's begin with covariance.
The formula for covariance is the following.
The covariance between two features X and Y is equal to one over N and the sum between x ie minus the
average value of x multiplied by y of T minus the average value of Y.
And we're going to break this down in just a second, but I want you to start building out an intuition
for covariance.
And keep in mind covariance can actually go from negative infinity all the way to positive infinity.
And you can see a visualization of the general relationship for something that has negative covariance.
So you can see it's kind of a negative slope there for covariance hovering around zero, there really
is no relationship and then covariance that is greater than zero.
You can see positive slope relationship and you can kind of tell from the formula here that it feels
similar to calculating the slope of a line.
But let's break it down a little further.
Before we continue a quick note.
If you were to Google the definition or formula for covariance, you are going to sometimes see this
as n minus one, and other times you may see it as just n like we're showing here to be a little more
specific and minus one is for sample covariance, while n is for population covariance.
But we'll discuss that in more details later on.
But right now we'll just show you the formula for population covariance.
So how do we break this down?
Well, we're looking at the covariance of data feature X with data feature Y.
For example, I could say, what's the covariance between the total bill and the tip?
Then you'll notice we're dividing by MN, where MN is the actual number of data points.
You should immediately know how this implies that X and Y need to have matching length essentially the
same number of data points, and that makes sense.
You can't really calculate the covariance between two data features that don't have matching data points.
So for example, if you went to one restaurant and had a couple of tips and then you went to another
restaurant that had a ton of instances of total bill, you can't calculate the covariance between those
because they don't actually share the same length.
And you should also notice that this feels a lot like the same calculation for an average.
So I want you to keep in mind that you're calculating some sort of average value because you're taking
a sum and then dividing it by the number of points.
And remember, this is the sum notation.
So what this is doing is it states to take the sum of the calculation on the right hand side for every
point in X and Y.
So you're going to start with x one and Y one, basically the first row of data, then x two and Y to
the second row of data, and you're going to go on for every single row of data.
So that would be every instance, for example, of total bill and tip all the way until you reach X,
y and Y.
Essentially that goes all the way to n.
I is just a placeholder for one, two, three, four and so on all the way to end rows or end data points.
Now let's take a look at what's happening inside this sum.
It's an interesting calculation.
Notice it's the difference between each individual point and the mean value for the data series.
Then you're going to multiply that calculation between X and Y together.
This entire formula should feel familiar if you've already seen the formula for just straight variance.
Notice how it's actually variance, but with another data feature.
So if you think back to the formula for variance, it was extremely similar except you squared it.
In this case, you're essentially doing variance between two features.
Thus the name covariance.
So now that we've briefly explored the formula, what information does Covariance actually report back
to us?
So if we take the total bills and tips, take those values, plug them into the formula, how do we
actually interpret this final value?
Recall that variance reports back the squared difference of the average value from the mean.
So thinking back to just variance, if we were to visualize this and calculate it, we're just talking
about what's the squared difference between the average value from the mean.
So visualizing this I think helps out a lot.
Imagine that we have a bunch of values and some of them are all of them fall between values one and
nine.
Notice we have no values for one, one value for two, three values, four for two values for five,
one value at seven and one value at nine.
So again, just talking about variance for right now, we'll expand this concept into covariance in
just a second, but I want you to visualize variance first.
Then it's going to become super easy to visualize covariance.
So thinking back on variance, what's the formula actually asking us to do?
Well, first off, I got to calculate the average value.
So I go ahead and calculate the average value here and notice it's hovering around five.
Then what I'm going to do is for every single point, I'm going to take that point, subtract the average
value, and then square that.
And if you start thinking about this visually, taking the square of this is actually the same thing
as taking the area of a square from that point to the average value.
So here what we've done is we've shown you those actual squares visually.
Notice how we're essentially constructing squares from each point all the way to the average value.
And again, this is for variance.
And then what you're going to end up having here are all those squares, and then you can end up summing
them up and then taking the average value.
And that's essentially showing you, Hey, for all these points, what's the average value of your distance
from the mean after you perform this squaring operation?
So that's variance.
And hopefully this visual, if you take the time, gives you a little bit of intuition behind what the
calculation is doing.
From a visual standpoint, again, you're taking the points, calculating the mean.
And then if you look inside the formula, what it's actually asking you to do is take each individual
point, go to the mean, and then take that difference and square it.
Remember, because you're squaring it, that's always going to be a positive value and then you're going
to take the average there.
So the same concept of area can actually be applied if covariance, it's just going to be in two dimensions
X and Y instead of a singular dimension, just X.
So let's take a look again at covariance and try to understand what's happening here.
Remember, this time it's going to be pretty much the same calculation and you're building out now a
rectangle instead of a perfect square and you're just doing it along two dimensions.
So you're going to start off again with X and Y.
So we can think of X feature as total bill and Y feature as tip.
And then what we need to do is start off by calculating the mean.
But instead of just a single mean like we did on that single line for X, now we're going to have two
means one for X and one for Y.
So I'll go ahead and plot that out.
And conveniently here it falls in the middle.
But obviously, depending on your actual data set, it may fall somewhere else.
But here I'm just showing you this blue dot is going to represent the value of x mean lined up with
a value at y mean.
Then you're going to go through each of your data points.
So let's go through a single data point X, Y, and y Y, and I go ahead and plot that out.
Now take a look at the actual equation.
This equation is just the area of a square in this particular case, and technically, it could be a
rectangle, too.
So before we had a perfect square, but this time we have a rectangle.
It's just going to be x, y minus x mean multiplied by y, minus y mean.
And again, that's just one side times the other.
So that's again just calculating a rectangular area.
This time it's in two dimensions.
So it's not going to be a perfect square because last time it was x, y minus x average squared.
And in this case we're just doing it for two dimensions.
But the same sort of visualization and intuition applies.
And then next, you simply look at the slope between the two lines to realize whether or not this is
going to be negative versus positive.
And therein lies one of the main differences between variants for a single feature versus covariance
between two features.
Notice this time the value could actually be negative.
Unlike last time with the square operation, it was always going to be positive and this is going to
be the same as the slope between the two points.
So notice how this is a negative slope, Meaning if you were to actually perform this mathematical operation,
you're going to have a negative value.
So we can kind of treat this area as a negative area for covariance.
So if it's a negative slope, like in this example, we can treat this total area as negative covariance.
And keep in mind, eventually we're going to do this for a bunch of rectangular or square values and
then you can add them up.
So, for example, at this point we can see that the slope between Z Y and the actual values of the
mean X mean and Y mean, that's going to be positive.
So I'm going to fill in this rectangle as having positive covariance and notice the slope basically
falls in line with that.
And now what you're going to end up doing is you keep doing this for all the points XY and Y until you
get to the last row.
And so you just keep drawing these rectangles in the same way you drew the squares for variance earlier.
And again, note how the formula indicates we're doing this for all X and Y points.
And then what you're going to end up doing is you're going to have a set of series.
So in this particular example, I'm showing you a series set where they're doing all those rectangles.
And in this case, a lot of them actually have a positive slope or a positive covariance.
Notice if you were just to look at the black points and kind of ignore those green rectangles.
For right now, the general trend looks to be like a positive slope.
So in this case you have a positive covariance and we can see for this particular example, the covariance
is 8.04.
There's also a correlation value, but we'll talk more about correlation later on.
But I want you to get this idea that when you see a bunch of points here with kind of this general positive
trend or positive slope, if you were to do this calculation of drawing those rectangles, which is
literally what the formula is asking you to do, you end up just summing those up and then taking the
average of the area of those rectangles, and that would supply you with the covariance value.
So hopefully this intuition and visualization really helps you understand what's going on behind the
scenes.
Let's take a look at a series with a negative covariance.
You can still see there are some points that have that positive slope, but the majority of them look
to have a negative slope.
So again, when you sum these all up and take the average, you end up getting a negative covariance.
And in this case the negative covariance is -0.119.
And again, we'll talk more about correlation later on.
You'll notice that the correlation is also negative.
And then four points where they're a little more flat.
You can see here we have two data series with almost zero covariance.
So essentially what's happening here is your positive rectangle area is going to start balancing out
with the negative rectangle area and that ends up summing to close to zero.
So again, we're going to take a deeper dive into covariance later on in this section.
But I want you to keep in mind that visualization of those positive and negative rectangular areas to
give yourself an intuition of how covariance is actually calculated.
And I would highly encourage you to literally just grab a piece of pen and paper and diagram this out.
And you'll notice the equation here is essentially the equation of the averages of the areas of rectangles.
So even though this formula at first looks really intimidating, it's basically kind of elementary school
arithmetic here.
It's just calculating the average areas of all these rectangles.
But it's being clever in the fact that it's using things like the average x value and the average Y
value.
So in review what covariance is, it's simply the sum of the area of these rectangles that you then
average out where the slope between the lines is going to indicate whether it's a positive area value
or a negative area value.
And then just again, simply average out the sum of those areas.
I would highly encourage you to try it out yourself on some simple examples of positive and negative
covariance series.
So just see if you can create your own little data set with two or three rows and then go by hand,
maybe plot them out just generally by hand and then see how that actually ends up reflecting the formula
of covariance.
So the formula for covariance can seem very intimidating at first, but it's literally just drawing
rectangles and then taking the average value of the area of those rectangles.
So now that we understand the intuition behind covariance, let's explore the related topic of correlation.
So what is correlation?
Well, correlation is actually very related to covariance and even uses the formula for covariance or
the resulting covariance in its calculation.
And as we learn about correlation, know how its possible range differs from covariance.
So remember that covariance could go from negative infinity all the way to positive infinity while correlation
has to be between negative one and one.
So essentially any data points that you feed into a correlation calculation are always going to spit
out a value that's between negative one and one.
So here is the correlation formula.
Pretty simple, right?
It's just using covariance.
So it's simply the calculation for covariance divided by the product of the standard deviations of the
two series.
So on the bottom here, you have the standard deviation of the x theta series multiplied by the standard
deviation of the Y series, and you're just taking that value and then dividing the covariance value
by it.
So why would you do this?
Well, let's expand this formula by showing the calculations for standard deviations and covariance.
So if I were to plug those back in, plugging in the formula for covariance and the formulas for the
standard deviations, I get something that looks like this the full formula for calculating the correlation.
Recall for covariance that X and Y are separate data series, meaning they could have very different
ranges of min and max values.
For example, the total bill could go from maybe $0 all the way to thousands of dollars, but the tip
itself maybe only goes from $0 to maybe hundreds of dollars.
So that's already an issue where tracking the covariance between two series that could have a relationship,
but their values are super different.
That's going to mean it's going to be hard to actually interpret the raw covariance number because again,
it can go from negative infinity all the way to positive infinity.
So instead it would be great to standardize the covariance so that it always has values between negative
one and one.
And I mentioned that the very start that businesses tend to just report correlation instead of covariance.
And that's the reason is because you always know it's going to be between negative one and one and it's
essentially standardized that way.
So it makes it way, way easier to interpret without needing to understand the ranges of the original
data feature values.
This can cause issues in trying to determine the significance of covariance values, since it can take
any value, as I just mentioned, which is why we use correlation.
So the formula for correlation actually standardizes this for us.
It uses the denominator to standardize the covariance between negative one and one.
So this is all to say if you're following this formula, you'll realize by using the mathematical trick
of dividing covariance, by the multiplication of the standard deviation between series X and series
Y, you end up standardizing covariance to always fall between negative one and one.
And when I say standardized covariance, I'm really just talking about correlation.
That's all it is.
It's standardized covariance so that it has to fall between negative one and one.
You can easily then interpret that number and you don't even need to know anything about the ranges
of X and Y.
So this makes it much easier, again, to interpret a correlation value rather than a covariance value,
which is why in the real world you pretty much always see correlation being reported rather than covariance
being reported.
And you can check out here what the difference correlations look like for different distributions or
joint distributions of features.
And notice that technically speaking, there's going to be some strange relationships between features
that sometimes have zero correlation.
So something to watch out for.
For example, you can see that we kind of have this ring or x looking values or something that looks
like a sine wave, and depending how you calculate correlation, when you standardize it, it's going
to end up being zero.
So it's just something to look out for.
But in general, when you're talking about correlation and joint distribution between two features,
you should think of one as perfectly correlated.
So one value goes with the other.
They're essentially equal to each other and negative one being perfectly correlated the other way.
So as one increases, the other decreases and then the values in between.
Notice again how certain data sets with interesting behaviors can still have a zero correlation value.
So there is helpfulness and actually visualizing a joint distribution along with reporting its correlation
value.
So we've now explored covariance and correlation and understand their ability to inform us of the relationship
between two data series.
And as I keep mentioning, you're most commonly going to see correlation use and organizations as the
metric being reported since it is standardized to be between negative one and one.
Okay, so let's take a deeper dive into covariance and correlation.