Everyone welcome back to this class modern Deep Learning in Python Deep Learning in Python.
Part two
in this lecture.
We are going to look at what I consider to be one of the most effective improvements over plain gradient
descent called momentum.
Personally I found it to be the 80 percent factor that can improve your learning procedure.
So how does momentum work.
I like the analogy of a ball rolling down a hill which is nice because during gradient descent we imagine
the errors surface as the bottom of a hill.
But I think today an even better analogy is moving something on a frictionless surface like ice.
So if you've ever gone skating before you know that you can easily glide along the ice very quickly
and without much force because the momentum just carries you the way you were going before you can imagine
pushing a box on ice and the box will move from one point to another quite easily because you can push
it once and it's going to slide.
That's exactly what happens in physics.
If there is no friction then an object will just continue to go in the direction it was already going
of course ice still does have some friction.
So eventually the box was slowed down and eventually stop as you'll see momentum in gradient descent
also slows down after a while to
to continue with this analogy.
Let's try to think of a situation with our momentum.
Now keep in mind I'm talking about gradient descent momentum here not physics momentum since the definition
of momentum and physics is a little different.
So imagine now that instead of ice you're trying to push a box in sand.
Now you can imagine that this is going to be very difficult because sand has lots of friction.
So if you push a box in sand it's going to go as far as you push it and no more.
If you want the box to move again you have to push it again.
That's like gradient descent without momentum each time.
If we want to move there has to be a gradient so that we can move in the direction of the gradient.
Now let's try to put these ideas into math.
Let's start with regular gradient descent without momentum.
So we have theta at time t is equal to theta at time T minus one minus the learning rate times the gradient
G.
And from this we can see that if the gradient is zero nothing is going to happen to theta it just gets
updated to its old value.
It doesn't change
now let's look at gradient descent with momentum.
We use the term momentum very loosely here since it has nothing to do with actual physical momentum.
So we create a new variable called V which stands for the velocity it's equal to new times its all velocity
minus the learning rate times the gradient.
And this is exactly what we talked about earlier when we imagine pushing a box on ice.
The G term is the effect of our pushing the box but the mew times V of T minus one term is the effect
of continuing to move in the same direction as we were going before.
Now we talked about how when you push a box on ice even though there's less friction and the box can
slide it's still going to stop eventually.
So we want the new V to take on only a fraction of the old V and so a typical value for Mew which we
also call the momentum term is zero point nine or zero point nine five or zero point nine nine.
Somewhere in that area
one thing you can immediately see is that if we plug in the expression for V of t we can get the update
for theta of T in terms of only the old theta the old V and the gradient G.
From here it's easy to see that if we set Mew to zero we just get back regular old gradient descent.
So what is the effect of using momentum.
Well as you know in all of our scripts we like to plot the cost per iteration.
And so what we usually see is that the cost converges to its minimum value much faster than if we had
not used momentum.
So using momentum is pretty great because it significantly speeds up training.
Now I want to discuss one more perspective on momentum before we move on.
This is what is typically taught in deep learning courses these days.
So in this perspective the problem that momentum is solving is if we have unequal gradients in different
directions.
So for visualization purposes let's assume we have two parameters to optimize the vertical parameter
and the horizontal parameter.
So the gradient in one direction is very steep and the gradient in the other direction is very shallow.
The idea is if you don't have momentum then you rely purely on the gradient which points more in the
steep direction than in the shallow direction.
And this is just a property of the gradient.
It's the direction of steepest descent.
So since this gradient vector points more in the steep direction than the result is we're going to zigzag
back and forth across this valley.
And that's a really inefficient way of reaching the minimum.
So what happens when we add momentum to this situation.
Well since in the shallow direction we move in the same direction every time those velocities are going
to accumulate.
So we'll have a portion of our old velocity added to our new velocity to help us along in that direction.
The result is that we get there faster by taking bigger steps in the direction of the shallow gradient.