In this lecture we are going to discuss gradient descent gradient descent is extremely important because
it's the algorithm that we use to train all of our models from simple neurons to neural networks to
convolution on their own networks and recurrent neural networks.
This one basic idea is the backbone of deep learning and in fact it can be used for other machine learning
models as well such as K Means clustering Hidden Markov models and Matrix Factorization.
Basically it is a general solution with wide applicability.
This lecture will discuss the basic intuitions about gradient descent but if you are an expert in mathematics
you might want to check out a rigorous proof that it works in a document of attached and extra reading
that X T.
It's called gradient descent convergence analysis.
First let's ask ourselves why do we need a gradient descent in the first place.
What is the problem we are trying to solve.
Recall that for problems such as linear regression we are going to come up with an error function also
known as a cause function or lost function.
Our goal is to minimize this error with respect to the model parameters call that W..
So how do we minimize L with respect to W
if you think back to your high school calculus studies you'll realize that you've already encountered
this.
Whenever we have a function and we want to minimize it or maximize it with respect to its arguments
what do we do.
We take its derivative and set it to zero.
Then we saw for the parameter in question.
So for example if you wanted to minimize 5 X squared plus three x plus one with respect to x you would
find the derivative with respect to x set it to zero and solve for x.
The reason why this works is because the derivative or the slope at the minimum and maximum of a function
is zero.
In other words I can draw a line at this point.
Tangent to the curve and it should be a horizontal line.
Now you might ask how do we know if it's a maximum or minimum.
There are ways to determine that but in machine learning we generally do not do that analysis.
We pretty much already know that the lost functions we are working with have minimums and not maximums.
Sometimes we may worry about saddle points or local minima but in general this is still not a problem
for the latest deep learning methods the reason why is outside the scope of this course.
But you are welcome to ask me about it on the Q and A
Okay.
This still doesn't answer the question why do we need a gradient descent.
Well here's the problem sometimes we can take the derivative and try to set it to 0 and solve for the
parameter.
But sometimes this equation is not solvable by the way I've maybe once or twice had someone asked me
why some equations are not solvable while my answer is.
If you think you can solve it then do it and show me the answer you can use your intuition about other
equations we do not know how to solve.
For example we can't solve.
Let's say the roots of a degree 100 polynomial it just simply is impossible
OK.
So what is the general strategy we use in mathematics when we cannot solve an equation.
The answer is we use numerical approximations.
One such example is finding integrals.
Sometimes you can't solve for an integral by hand.
I think anyone who is taking calculus knows that solving some integrals can be quite challenging and
for others literally impossible.
So what do we do.
Well instead of trying to solve the equation by hand we just draw a little trapezoid underneath the
curve and we find the area of those trapezoid.
That will give us an approximation for the area under the curve.
OK so that's the analogy.
When we can't solve something analytically or by hand we resort to a numerical approximation methods.
So what's the idea behind gradient descent.
The idea is if I take the gradient of the function with respect to the parameter W and I take small
steps in that direction the value of the function will decrease.
Provided that my step size is small enough eventually as I keep taking these very small steps I'm going
to get closer and closer to the bottom of the curve.
And eventually I will converge to an answer by converge I mean that when you get near the bottom your
steps will become so small that they will become insignificant.
So w doesn't change as you take more steps at that point.
You have found the value of W that minimizes your function.
So what does this look like in code.
Basically it's just a for loop before the loop we initialize W to some random value then inside the
loop we continually update w using the gradient descent formula
so here's an example of gradient descent at work.
Let's say we want to minimize w squared.
So that's our last function.
The derivative is 2 W.
So inside the loop we iteratively apply the equation W equals W minus eight times two times w.
Now let's look at some code I've pasted that does this.
As you can see I've initialized W to 5.
I created an empty list called L where I'm going to store the loss at each iteration so that I can plot
it later.
Now in a loop I run my gradient descent update one hundred times and it each time I append w square
to l finally I plot out we can see that eventually the value of the function goes down to zero.
It's a minimum value.
The value of W that we find at the end is also zero because this is the value of W that makes the original
function zero.
Now you will notice that there is some hyper parameters that can't quite be ignored in particular how
many iterations should we run the loop for.
This is known as the number of epochs of course we must have a number of epochs high enough so that
the cost function converges but it must not be too high.
Otherwise we'll waste a lot of time where w simply doesn't change how small should our step size be.
I've used the Greek symbol Ada for this and we call it the learning rate the learning rate must be small
enough so that the cost function doesn't explode but it also must be large enough so that the learning
process doesn't take unnecessarily long.
Both of these hyper remedies are values that we must choose using trial and error.
There are a few more advanced methods such as Bayesian optimization that you could use also but they
are outside the scope of this course.
Normally for students of machine learning doing experiments manually and gaining intuition through practice
is the best approach.