In this lecture we are going to discuss the means squared area in depth the goal of this lecture is
to ensure that you understand the means squared error from a probabilistic perspective.
This helps to give some theoretical backing for the mean squared error and it will help prepare us to
discuss the cross entropy loss as well as a side note.
I just want to remind you that there are multiple names that we use interchangeably.
So if you hear error or loss or cost or objective.
These are all generally referring to the same thing.
So we might say cross entropy error but we also might say cross entropy loss they mean exactly the same
thing.
First let's just make sure the mean squared error makes sense in the first place.
Why is it squared.
Well the idea is that we would like the error to always be positive.
Imagine this scenario one of your predictions has error plus 2.
The other prediction you've made has error minus 2.
If you add these two errors together you would get plus two minus two which is yes zero.
Of course this is not zero error because both of your predictions are off by 2.
They should not cancel each other out.
So squaring the error is a nice way to make it positive
but this still doesn't really explain why we want the Square Mile at the absolute value.
Why not some other method of making the difference positive.
In fact it's perfectly OK to use the mean absolute error.
We just don't normally refer to that as linear regression.
So when we say linear regression usually we are talking about a linear model using the mean squared
error.
OK so let's dig a little deeper on why we would want to square the error to understand this discussion.
You have to be familiar with maximum likelihood estimation what you would normally learn in an introductory
probability course.
Let's say for example we would like to measure the heights of all the students in our class and we decide
to model this as a Gaussian distribution.
As you recall the Gaussian distribution is the typical bell curve.
It has the probability density function that you see here which will become important momentarily.
We know intuitively that the best way to estimate mu let's call that you have is the sample mean of
all the heights we have measured.
The question is why is this so
here's how we solve this problem.
There is actually a method to find this view.
This method is called maximum likelihood estimation.
As I mentioned earlier.
So what are the steps.
First we create the likelihood a function let's call that El.
Unfortunately we use the letter L for a lot of things and deep learning.
So please don't confuse this L with anything else we've discussed.
The likelihood is the multiplication of the probability density function at each of the x values we've
collected.
We consider this to be a function of Mew.
So here MU is the variable and the X's are constants because they are just numbers.
They are the heights of the students that we have measured in our class.
The next thing we would like to do is maximize L with respect to mu.
Let's think about why we would want to do this just for intuition.
Imagine that you measure someone's height x.
Now imagine Mew is very far away from X in this case the probability of X is very tiny.
We might think that X is not from this probability distribution because P of X yields a very small value
but what if Mu is equal to x.
In this case p of x yields a very large value.
Thus we want p of x to be big if it is to be representative of the data we've collected
so it's the same idea for L we want l to be big so that MU is close to all of the data points x that
we have collected.
So how do we maximize L with respect to mu.
Luckily we can just ask our old friend calculus.
Normally the process would be taking the derivative of L with respect to MU and then set this derivative
to zero and solve from you.
But for probabilities.
Taking the log of L first usually leads to a more easily solvable derivative.
The reason why this is allowed is because the log is a monotone likely increasing function.
Maximizing L is the same as maximizing the log of all the value of Mu that maximizes L is the same as
the value of Mu that maximizes lago.
You can prove this to yourself intuitively by taking any two numbers a and b if a is bigger than B.
It is also the case that logic is bigger than log B
Okay so I'm not going to bore you with the calculation but as always you are welcome to do it yourself
on paper using your trusty calculus skills.
You should end up with the derivative we see here.
So if you want please pause the video and make sure you can take this derivative yourself.
OK so what do we get if we set this to zero and sell from you.
Well you can see that we get back our original answer that Mew is the sample mean of X..
This is called the maximum likelihood estimate of Mu because in order to find it we maximize the likelihood
here's what's important to notice about the log likelihood if you notice the log likelihood has the
same form as this expression where I've replaced the constant terms with C1 and C2 you'll realize that
it actually doesn't matter what values C want to see to have they don't affect the answer we get eventually
that MU is the sample mean of X. Although we do recognize that C is positive C one is also positive
given our old expression but it actually doesn't matter what it is at all.
In other words they are just superfluous constants.
So what happens if we set these constants to convenient values such as 0 and 1 over n
where we get an even more interesting expression.
Little L. The thing we want to maximize is the negative sum of X Y minus Mu squared
one thing you may recall from your study of calculus is that maximizing something is the same as minimizing
the negative of that thing.
In other words if I want to maximize negative x squared I should get the same answer as minimizing X
squared the value x equals zero is the answer in both cases.
Hopefully you can see right away that this is obvious
at this point you probably get the gist of what I'm trying to say maximizing the likelihood is the same
as minimizing the mean squared error where the error is the difference between X Y and Mu
so how can we relate this back to problems such as linear regression.
Take a look at these two error functions.
The first error function comes from our earlier problem finding the best Mew to model the heights of
the students in our class as a gaussian the second error comes from linear regression.
Make the Y hats close to the wives.
How can we relate the second problem to the first problem.
Well if we say that the Y eyes are samples drawn from a Gaussian distribution with a mean is y hat ie
then minimizing this mean squared error is the same as minimizing the negative log likelihood and that
of course is the same as maximizing the log likelihood which is the same as maximizing the likelihood
finally.
One thing we can do is instead of saying that the mean of the distribution is why have we can simply
say that Y minus y have is a gaussian centered as zero equivalently.
This is saying that when we use the squared error we are making the assumption that the error of our
model with respect to the data is Gaussian distributed with mean zero.
This formulation helps us to put a probabilistic backing on the error function and it will also help
us make sense of the Cross entropy loss for classification.