Hey everyone and welcome back to this class modern deep learning and Python Deep Learning in Python.
Part two
in this lecture.
We are going to discuss variable and adaptive learning rates.
So far we've looked at momentum as a modification to vanilla back propagation that can greatly speed
up training.
And from my perspective I find momentum to be the most impactful.
If you compare the last per iteration for a standard gradient descent versus gradient descent with momentum
the difference is huge.
At the same time momentum is nice because you don't really have to play with the hyper parameters that
much just go with zero point nine.
And it's probably fine.
So momentum is often my go to at the same time.
Some of these adaptive learning rate techniques are very powerful.
So let's have a look.
The first thing we'll talk about is variable learning rates or in other words lending rates as a function
of iteration or time.
Sometimes people call this learning rates scheduling.
They are very simple but they give you a lot to play with.
So the first one we're going to look at is called step decay.
Essentially what we do is periodically say every 100 steps we reduce the learning rate by a constant
factor.
So for example we divide by two or have it each time.
If we plot it you can see that it kind of looks like a staircase
a second method is called exponential decay in this method.
The learning rate follows an exponential curve.
The third method is for the lending rate to decay proportionately to one over time.
You can control how fast or slow the lending rate decays by changing the proportionality constant.
Notice how the drop off is slower than exponential decay.
So what do all of these have in common.
Well you can see that they all decrease the learning rate with time.
Why would we want to do that.
Well generally speaking when we initialize the weights of a neuron that we're randomly they're going
to be very far from the optimal weights.
So it's good to start with a large learning rate so that we can take bigger steps towards the goal.
This is the motivation behind momentum as well.
We want to pick up speed by accumulating past gradients because we know that if we are very far away
from our goal then those gradients should be large.
But when we get close to the goal the gradient is going to shrink.
In fact by definition the minimum of a function assessor takes a gradient of zero.
That's how you solve for a minimum of a function and calculus.
If you recall you finally derivative and then you set it to zero and then you solve for the parameter
in question.
So why might we want to slow down when we get close to the minimum.
Well when you close to the minimum and you take too big of a step you're going to overshoot.
And so what ends up happening is you just end up bouncing back and forth.
In fact if you're learning rate is too large you'll just bounce right out of the valley and your laws
might actually increase.
So in order to reduce all this bouncing around we would like to take small steps.
Here is another technique that people use which is related to these variable learning rate techniques.
Sometimes researchers and machining practitioners will actually sit there and babysit the neural network
training process so they'll do a couple of epochs and see how it goes if learning is happening too slowly.
They might increase the learning rate if learning is starting to even out they might decrease the learning
rate but you have to be careful because you're not guaranteed to have a nice monotone ugly decreasing
curve.
Sometimes you might get stuck in a relatively flat portion of the Arab surface but this could just be
temporary.
And if you're patient enough you might end up getting a steep drop.
But at the same time this is not guaranteed as with most things in machine learning.
The answer is that behavior is data dependent waiting longer is desirable.
If you have the resources because then you get a full picture of the learning process.
So manual learning rate scheduling is also an option for you to choose from
now.
Also important to think about what else to all these methods have in common.
Well all of these methods add new hyper parameters to your list of things to optimize.
In fact choosing between each of these methods can also be seen as a hyper parameter optimization.
So in one way knowing about these techniques can help you but from another perspective it adds more
work to your plate
the next couple of techniques I want to talk about are what I like to call adaptive learning rate techniques
because they adapt to the training data that you've seen so far.
The first one I want to discuss is called the eighth grade.
The basic idea is this we can't expect the dependents of the costs on each of the parameters to be the
same.
In other words in one direction the gradient might be really steep but in another direction the gradient
might be really flat.
So perhaps it may be beneficial to adapt the learning rate for each parameter individually based on
how much it has changed in the past
so an undergrad what we do is we introduce a variable called the cash each parameter of the neuron that
work has its own cash.
So for example if you have one way matrix of size three by four then you'll also have a cash matrix
of size three by four and the same thing goes for your bias vectors.
The idea behind the cash is that it's going to accumulate the squared gradients because we're squaring
the gradients.
The cash is always going to be positive and because each parameter has its own cash then if one parameter
has had a lot of large gradients in the past then its cash will be very large and its effective lending
rate will be very small so it will change more slowly in the future.
On the other hand if a parameter has had a lot of small gradients in the past then its cash will be
small so it's effective lending rate will remain large and it will have more opportunity to change in
the future one minor detail is that we usually add a small number Epsilon to the denominator to avoid
dividing by zero.
Typically this is set to 10 to the minus eight or 10 to the minus 10
one important point to stress about eight grand is that everything we're doing is an element y's operation
so each scalar parameter is effectively updated independently of the others which makes sense.
So you can look at the formulas we presented as scalar updates which apply to all the parameters or
you can think of one huge parameter vector that contains all the neuron parameters.
And then each of the operations is an element y's operation
this next technique builds on the fact that researchers observed that 80 grand decreases the learning
rate too aggressively.
So the learning rate would approach zero too quickly when in fact there was still more learning to be
done.
This technique is called Our M.S. problem and was invented by Geoff Hinton and his team.
So the way it works is this the reason it agreed decreases the learning rate too quickly is because
the cash is growing too fast.
So in order to make the cash grow less fast we actually decrease it.
Each time we update it we do this by taking a weighted average of the old cash and the new squared gradient.
We call this way the decay rate and you can see that the two weights add up to 1.
We'll be looking at this form again and again throughout the course.
So don't worry about knowing all there is to know right now.
Just know that we'll be taking part of the old cash and part of the new squared gradient and adding
them together to get the new cash typical values for the decay rate.
Our point nine nine point ninety nine point nine nine nine nine and so on again the intuition for these
choices will be discussed in later lectures so by doing this we say that we're making the cash leaky.
And the reason it's leaking is because you can imagine that if we had zero gradient for a long time
eventually the cash would shrink back down to zero because it would be decreased by the decay rate on
each round
one obstacle.
I realized sometime after this course was originally released is that there is actually some ambiguity
in the arms prop in eight grad algorithms.
Specifically I discovered this in the context of ARM as prop so one thing that's not specified in the
arm is prop.
Algorithm is the initial value of the cash.
Now one might automatically just assume that zero is a good value.
And I did this as well but this actually has a very strange effect.
Let's suppose you choose the decay rate of zero point nine nine nine.
Then your initial update for the cash will be zero point zero zero one times G squared.
So you're taking one one thousandth of the squared gradient so disregarding Epsilon for the time being
your effective learning rate actually becomes very large due to the fact that the denominator is very
small.
What you'd have to do is compensate by making your initial learning rate smaller than usual
one solution to this is to set the cash to one instead.
By manipulating the arms prop update a little bit we can show that this gives us approximately what
the update would be if we weren't doing our mass prop at all.
Or in other words the effective learning rate is approximately equal to the initial learning rate divided
by 1.
Now you might ask Well which one is the correct way.
And we can't really say because it was never specified what I can tell you is that there are some very
major packages that have implemented them both ways.
So it's not as if one is more correct than the other.
For example intensive flow the cache is initialized to 1 in carries the cache is initialized to 0.
In fact it's a great exercise to try and implement these adaptive learning rate techniques.
Then compare your implementation to these official implementations to check if your implementation is
correct.
If it is correct then your last per iteration should match what the library gives you.
And so I've done this in the file AMAs prep test up high in case you want to check it out.