Welcome back everyone to this section on using your regression.
You've made it to the last section of the course.
So a huge congratulations for all the learning and work you've done so far.
In this section, we're going to focus on directly applying statistics to data sets to create models,
and we're going to be using one of the earliest and most common techniques known as regression.
So we're going to be covering a lot of different topics that in some form, one way or another, are
related to regression, such as scatter plots, correlation coefficients and how they relate to the
residual and the coefficient of determination, root mean squared error and more.
But for right now, let's focus this introduction on regression and why it could be useful.
Formally speaking, regression is a statistical technique that uses data sets to create explanatory
models.
These explanatory models can then be used to help predict future outputs of dependent variables based
on inputs of independent variables.
Even though regression models have been studied since the 1700s in astronomy, they are actually still
extremely useful today, and they're often easier to understand than more complex statistical models
that fall under the umbrella of more advanced machine learning.
We can think of regression as the connecting piece between the world of classical statistics that we
just learned about in modern machine learning techniques.
It's often one of the first models applied because it directly produces coefficients that can be interpreted,
which is an extremely useful features that not every machine learning technique actually provides for
you.
Now, you've probably already seen the capabilities of regression models in practice.
For example, imagine you started a company to buy or sell houses based on features or variables such
as the area of the house, the number of bedrooms, the number of bathrooms, etc. So you have a bunch
of features and variables about the house, and then you're trying to predict what's the appropriate
price for the house.
Regression models not only provide those direct prediction properties such as modeling that approximate
price, a house should sell it given its features or variables, but a regression model also informs
you of the effect of each variable through the use of what are known as coefficients.
So that way you can actually understand which features like the number of bedrooms is most important
for predicting that dependent variable the price of the house.
And you're probably already familiar with a very simple linear regression function such as Y equals
M x plus B, where M is the slope and B is the y intercept where the line crosses the y axis.
Let's start with this very basic concept of regression and a linear fit and then expand its use to a
real world data set.
Put simply, a linear relationship implies some constant straight line relationship.
The simplest possible being Y is equal to X.
Technically, you could also say y is equal to a constant, but let's just keep Y is equal to X for
right now.
So you notice I have three data points and they all fit.
Y is equal to x, I have x equal to 1 to 3 and Y equal to 1 to 3.
So if I were to pair them up, it'd be one, one, two, two and three, three.
Now, what I could also do then is based on these data points, I can build out a relationship known
as a fitted line.
I'm fitting a line to these data points.
Now, what this implies is if I get some new incoming data like X, I can predict its related y value.
So if X is equal to 1.5, then because I have my fitted line equals to y equals x, then for the input
x equal 1.5 I get the output y is equal to 1.5.
Now, that was a very simple example.
But what happens with real world data that doesn't actually fit nicely all along a straight line?
What do we actually draw this line?
Do I draw it up here or do I draw it somewhere here in the middle?
We need to figure out some sort of metric that allows me to understand which line best fits the data.
Fundamentally, we understand that we want to minimize the overall distance from the points to the line.
So if I draw a line like this, I can actually measure the distance between those points to the line.
And this is known as the residual error.
That means some lines are going to clearly be better fits than others.
You can see for this line, it looks like the distance between the line and the other points is less
than that previous line I showed.
We can see as well that the residuals can be both positive and negative, since the line itself can
be below some points but above others.
And that idea of being positive and negative for the error term is going to be important when we try
to evaluate the actual overall error of the line.
So what kind of mathematics can we use to try to figure out the line of best fit?
We can use ordinary least squares, which works by minimizing the sum of the squares of the differences
between the observed dependent variable that is, the values of the variable being observed in the given
data set and those predicted by the linear function, which is essentially your y output.
So let me show you how you can visualize the squared error that you're trying to minimize.
Notice that I have my real data points as blue points.
Then I have a line of that.
I'm trying to fit in orange and a distance between those points in a dashed red line.
I'm trying to minimize the squared error.
Which means I can just square the distance between my points to the line.
And that's area of all those squares is what I'm trying to minimize.
Having a squared error will help us simplify our calculations later on.
When setting up a derivative recall that I mentioned, the error could be positive or negative, but
if you square that then everything becomes positive, which is very convenient mathematically.
So now it doesn't really matter whether the line is above or below a point.
The act of squaring it is going to make it positive.
Let's continue exploring ordinary squares by converting a real data set into mathematical notation,
then working to solve a linear relationship between features and a variable.
So right now we know the equation of a simple straight line Y equals m x plus b m is the slope, B is
the intercept with the Y axis.
We can see four Y equals m X plus B.
There's only room for one possible feature X ordinary squares will allow us to directly solve for the
slope M and the intercept B.
Let's explore how it could translate a real data set into mathematical notation for linear regression.
Let's imagine that I'm actually trying to build out a linear regression model for predicting the pricing
of a house.
Well, I have a bunch of features that is multiple features of the house shown here in blue, like the
area in square meters, the number of bedrooms and the number of bathrooms.
And what I'm trying to do is estimate a target output, which is the Y output of my linear model function.
And in this case it happens to be the price of the house.
So I can begin to translate this into generalized mathematical notation by thinking of those independent
variables or features as x or a matrix and Y, which is just that single vector or column of the price
that is the output I'm trying to predict.
So I could then start using more mathematical notation and actually build this out.
So I just have a bunch of X's.
And then why's and this is known as linear algebra notation where you have a notation indicating what
column and row you belong to for the original data set.
We don't really need to worry too much about that specific notation right now, but I do want you to
be aware of it.
So now what we can do is build out a linear relationship between the features X and the label Y.
So I'm going to reformat this to look like a Y equals X equation.
Notice I basically just swap the order from the X column being first to the Y column being first.
Now each feature should have some beta coefficient associated with it.
So what I'm going to do is for every general feature, remember it was area number of bedrooms and number
of bathrooms.
I'm going to attach some beta coefficient so I will have beta not or beta zero attached to zero plus
beta one attached to x one and so on and so on.
As a quick note, I formatted my notation to start at one, but realistically, if you're starting to
dive much deeper into regression, you're probably going to start your notation at zero, because that's
how a lot of programming languages work internally.
So I should point out that this is pretty much the same thing as the common notation for a simple line.
I just have more X's that I'm dealing with.
So this is stating that there is some beta coefficient for each feature to minimize the error.
So I can also express this equation as a sum.
Basically, what I'm going to say is y hat, which is another way of saying my predicted y value or
predicted price of a house is equal to a particular beta coefficient multiplied by the first feature
value plus another particular beta coefficient multiplied by the next x value, and so on and so on
for any number of features.
And I can then reduce that notation y to just be a sum of beta times x.
And again, note that the y hat symbol displays a prediction.
There is usually no set of betas to create a perfect fit to Y.
That's what we're actually saying y hat instead of just y because I'm not going to get a perfect fit.
And I don't want to imply by stating y that it's going to perfectly predict all the values y hat basically
tells you, hey, this is an approximation of what y could be.
It's not going to be the exact value.
Otherwise that would imply that the line is going to touch every single point.
So again, I have my line and I'm trying to minimize the distance between the line and the points.
And y hat is going to essentially be an estimation of the output.
So let's imagine that I have a feature X.
What is the beta coefficient actually do?
If I think of this in terms of y equals m x plus B where M is now really just a beta coefficient.
All this is really doing is a multiplying factor for a slope.
So in this very simple case where I have one feature X and then I'm multiplying it by a beta coefficient,
then I'm really just moving that to some particular slope and we're going to do this across multiple
dimensions.
Obviously I can't draw more than two dimensions on a2d slide, but hopefully you get the idea that it's
just adjusting that slope across each dimensional feature.
And then y hat is going to be your actual predicted value.
Remember, y hat itself is not the precise value of Y.
That would be the actual blue point.
Y hat is our predicted value of y given an x input multiplied by the beta coefficient that we calculated.
So we have now created a simple model that can help predict the price a house should be bought or sold
at given a historical data set.
We simply need to solve for the beta coefficients which can be done by hand for only very simple situations.
Technically speaking, this is usually done of computational techniques using something known as stochastic
gradient descent.
So I should point out at a certain point regression just really can't be done by hand.
But we're going to try to keep it as simple as possible for this section of the course that we can really
get an intuition of what's happening.
The important thing to realize here is that we would have a set of PN equations with known X values
and known Y values for each row.
I.
The beta coefficients can then directly inform us of the strength of each variable in causing the output
y.
Let's continue to learn and explore some of the basic concepts of actually solving for those coefficients
for regression.
We'll see you at the next lecture.