Now, let's talk about optimizes and learning rate schedules.
These are methods that we actually use in addition to basically a more advanced gradient descent methods
that we use to find the optimal weights.
So let's get started.
So remember optimizes of basically methods like the gradient descent method, which we discussed.
It's an optimization algorithm, which is why we call it optimizes that allow us to find the lowest
weights to all the bits that give us the lowest loss.
So in the last section, we covered gradient descent.
So now we'll take a look at some other optimization methods that are more advanced that can offer sometimes
better performance or faster convergence when compared to stochastic gradient descent.
So before we go on, let's take a look at some of the problems with the standard stochastic gradient
descent method.
So firstly, we have to manually set the living rate or if we're not using it manually set lending rate,
we have to decide on learning which schedule which we'll talk about shortly.
So that that basically gives that's a problem, because now you kind of have to know what, what values
use in advance.
We can try a value in a way some time and try and evaluate with some time.
So it's not the best way to actually pick a learning rate.
So using the same learning rate for the parameter, it's often times allows us to jump over or get stuck
in, look of animals and jump over the global minima at times if it's too big.
So to solve some of these problems, a lot of researchers have developed basically extensions to suggest
a good idea to send.
All of these basically operate on the same underlying principles.
They're just different ways to make the update differently.
OK, so let's take a look at the first advanced feature of SAGASTI gradient descent that we can use.
This one is called momentum.
So one of the issues with Sagasti gradient descent was that areas where we'll clean is much steeper
in one direction, which basically doesn't work too well in that case because it sees it as a bad thing.
So because it's got a big spike, basically, it's what I'm saying, so it's going to not want to move
in that direction.
So this results in this Segretti a descent oscillating around is for making very little progress to
actually finding the minimum point.
Now what momentum does?
It increases the strength of the updates four dimensions whose gradients switched directions and bit
with big changes.
So this dampens oscillations.
As you can see with momentum, here is a lot less oscillations and it gets fitter.
It gets closer.
Sorry to the minimum point here.
Typically, momentum, we use a value of point nine that usually gives us some good results.
Next, let's take a look at Nazarov acceleration.
So one of the problems introduced by momentum was overshooting the local minimum.
So what nature of the acceleration does is basically it provides like a corrective update just so that
we don't overshoot the local minimum.
This is an illustration of how it works.
So these are the updates.
This is the initial sorry.
This is the initial stochastic gradient descent.
This is Néstor of sort of with momentum, and this is an order of correction in red.
So this is the final update in Vector to form here.
So this is a look at the other optimized this that we can use.
I'm not going to go through all of these in detail.
However, I will say that Adam is quite effective in a lot of training scenarios.
I use Adam quite often as well as I do use the custom gradient descent a lot with momentum.
I don't often use Néstor of correction, but I do use momentum quite a bit and I get tend to get good
results when that doesn't work.
I use Adam, but it depends on your dataset and your model that you're, you know, doing, so you can
pause it and read these if you would like.
I'm not going to discuss them in too much detail because they get quite technical.
So it's not that important that you know this anyway.
However, here's a visual comparison of them.
So you could play this animation here, and that shows you basically hold on different methods.
Take to convince you can see stochastic gradient descent.
The red one is taking quite a long time, whereas let's see who is this thing.
The moment momentum one was first, perhaps or at adulthood.
But it's a good animation to see this other one here shows you a similar thing as well.
Consider all trying to find the global minimum, which is down here.
However, you can notice CTD Stochastic or did descend got stuck into local minima here.
It never found its way down slope like the others did.
So you can see why having an optimizer that has that has the ability to change the learning rate of
deaths is very important.
So now let's move on to learning read schedules so you can see sometimes you may not want to use an
algorithm that uses an advanced optimizer.
So for whatever reason, maybe you're very specific with what learning rates you want to use for your
experiment.
So in that case, we can use stochastic gradient descent with something called learning, which schedule
which sets the learning rate for each epoch.
It's basically a simple lookup table to see it epoch one to use this value equal to that value.
So usually you progressively start at larger learning rates and then progressively get smaller over
time.
We can use this because if ordinary, it is too high.
You tend to overshoot minimum points.
So it's a good way to basically adapt your neural network learning process if you wanted to have that
precise control.
So the with schedules of directive very simple to implement paid too much carrots and TensorFlow all
incorporate different ways to do it so you can manually, such as you can have a key parameter.
So it is quite configurable and we'll get to this shortly.
And yet in future lessons.
So we'll stop there.
And basically, this concludes all of the background knowledge you need to understand for training you
on CNN.
However, I understand that this is a bit overwhelming.
It's been roughly an hour of these lessons, and you may be quite confused still, which is understandable
because it's a fairly deep topic.
Pardon the pun.
So what I'll do, I'll have a review session.
This is going to be like a five to 10 minute video where in this next in the upcoming section where
I go over briefly, all of the theory we have we had in the previous slides.
So hopefully that can help you fit together all of the knowledge you just learned so that you can understand
CNN so much better.
Thank you.