Hi and welcome to the section on gradient descent, gradient descent is basically the algorithm or technique
that we use to find the optimal rates in the training process.
So let's take a look at something here.
Remember the lost function we discussed earlier?
How do you find the lowest loss?
How do you get the gradients to get the lowest loss?
We've already seen a back propagation is a process by which we used to did the individual beats or gradients.
And basically our goal here is to find the best set of weights where we can get the lowest loss possible.
So no, this is important.
It is to distinguish here the method by which we achieve this goal.
That is a method in which we use bat propagation to lower the loss, to change the weights, to lower
the loss is called gradient descent and gradient descent is basically all about us finding the optimal
way it's where the loss is.
Lewis.
And this is a very basic to the example of what I'm talking about.
Imagine this is the loss halo sequel.
Bad, no loss.
Equal good.
And this is the point we want to achieve with the way it's due to the values of the weights and we want
to end up here.
We want to have a neural network with those weights.
So let's take a look at gradients now.
Gradients are the derivative of a function.
So basically, it tells us the rate of change of one variable with respect to another.
In this case here.
Remember, this was error loss and this was a it's where you just just said it.
So this tells us how much error changes with respect to changes in the weights.
So let's take a look at what this means to a positive gradient means loss increases if weight increases
and a negative gradient means loss decreases if weight decreases.
That's in this case here.
So at point, a single look at this guy point eight, if he's moving to the right, that increases.
Oh, it's because you can see the weights go up.
Let's move in the right direction.
And it also decreases our loss.
That means it's going to have a negative gradient here at Point B. Though let's take a look at this
moving to the right increases or weights and increases over loss, and that's going to be a positive
gradient in this side because it's going up.
So therefore, the negative or gradient basically tells us the direction we want to be moving.
In this example.
So let's talk a bit more about gradients.
So the point at which gradient to zero means that small changes to the left and the right doing change
loss.
This is a good thing, but it's also a bad thing.
And let's take a look at why.
Imagine we're at point C now, so you can see it.
Point C the small change, the left or a small change to the right doesn't really change the loss all
that much.
This means that it's stuck in a local minima.
So this is a bad thing, though, because we want to end up here.
Remember, look at this point here.
He has a much lower loss than a potency.
However, inadvertently during the training process, if our living rate, which we'll discuss in the
next upcoming slides at the learning rate is set to small our neural, our gradient can get stuck,
our function can get stuck here and it's never going to change, is we?
It's anymore.
It's going to converge at this point, and it's going to see this is the best I can do when in reality,
this is the best you could have done.
So let's take a look at what local and global minimums are.
So just to expand in the previous slide, these points in this graph here, these are the local minimums
of this function.
However, this is the global minimum right here.
This is the point where we want to get to.
This is a point where it has the lowest loss.
So think about really innocent.
Remember the gradient descent?
There's a method by which we use that propagation to get the best tweets.
It's basically a system in which we use to back propagation algorithm to just find a way to get to the
lowest points, the lowest loss and get the weight to that point.
So imagine this example here.
Imagine being a really tiny person and you're traversing down the slope of this old rough border.
There'd be peaks and valleys and troughs because if you're is quite small, it's an old rugged type
will, then you're going to probably see a lot of like bumps and cracks in it.
So how do you know when you're truly at the bottom because you're so small at this point, you can possibly
take large steps so you didn't get stuck in a valley.
However, you risk jumping over the bottom global minimum point.
So let's take a look at that here.
Step size is important.
That's step size can tell us basically, if you move in this direction.
That's a good thing.
You'll end up there at this point.
However, if you move a step size of this size again, you end up here.
So, no, you're left thinking that this is this is the global minimum.
When?
In reality, it's not.
So if you were to move backwards, oops, yeah, and if you were to move backwards, you end up in this
same situation.
So what we need here is a larger step size to find a global minimum.
However, step size can be too large, and you can sometimes hop over the global minimum as well.
Let's watch this animation again.
So remember, we talked about land of land, there was a lending rate, just basically it's just this
adjusts to step size.
So we're having a landing factor here when we have detailed weeds.
This allows us to control the magnitude of how much we jump when updating our wits.
You want to take small steps, but not too small steps, because then it takes forever to converge and
you risk getting stuck in local minimums.
However, you don't want to take large steps so that you miss the global minima entirely so you can
see it.
Finding the right lender is a bit tricky.
However, we'll get to that point afterward.
Let's talk about some gradient descent methods.
So a naive gradient descent.
This just passes the entire dataset should that work and updates to its of the end.
It's computationally expensive and slow, and it's not always a good thing.
So that's always, not always a good thing.
But stochastic, gritty descent updates to weeks after each sample is an input image is followed propagated
to the network.
So in a way, you can see this been better.
It's not going to be as computationally expensive as passing through the entire thing.
However, this leads to a lot of noisy, fluctuating lost values, and it's also very slow to train
to so many batch gradient descent, which is what we do in practice in the real world.
This is the method that works best.
Combines board methods.
It takes a batch of data, let's say, 16 images and info.
It propagates it all and then updates to gradients at the end.
This leads to faster training, convergence and convergence and global minima.
So we'll stop there now.
This is a system that you know about is that typically it to 256.
However, they can be one one basically doing stochastic gradient descent at that point.
So it's best to use the biggest bad size you could use that your RAM will support the GPU memory will
support.
So, no, we're going to take a look at optimizers as well as looting read schedulers.
So stay tuned for that lesson.
Thank you.