In this lecture, we'll be looking at another way to modify gradient descent using a technique called
adaptive moment estimation or Adam for short.
Adam is a technique that deserves its own lecture because it is the go to method for optimizing neural
networks today.
That is, people most often use Adam as the default choice without really considering other options.
The reason for this is simple.
Adam tends to work well and it tends to work well with the default settings for the learning rate,
momentum parameter and so on.
In other words, Adam is robust.
You can use it and not have to worry about tweaking the knobs to find values which are exactly right,
since what you have by default is probably OK.
In addition, Adam is somewhat of a successor to arms.
Both were developed at the University of Toronto Arms prop was developed by Geoffrey Hinton while Adam
was developed by Jimmy Ba, who is a PhD student of Hinson's at the time.
In fact, Jimmy Ba had three PhDs supervisors, all of whom are prominent machine learning researchers,
and he is now a professor himself.
Now, one way to think of Adam is that its arms prop with momentum.
This is because it takes the same idea of having an adaptive learning rate, but it also adds momentum.
This might be a little confusing since libraries like Tenzer Flow also provide you with an option to
add momentum to arms prop.
However, these are slightly different.
So the key point is Adam is like arms with momentum, but it's not exactly the same.
Let's start by reviewing what we have seen so far.
Specifically, let's consider two helpful ways that we can improve vanilla gradient descent method.
Number one is to use momentum.
An object in motion stays in motion.
Method number two is to use adaptive learning rate methods like arms prop.
Note that I'm using slightly different symbols than what you are probably used to.
This is just to set us up to learn Atem, which uses different symbols by convention.
The parameter update for momentum proceeds in two steps.
First, we update the momentum value, which I'm going to call em in this lecture.
It's equal to some proportion of the previous momentum, minus the learning rate times the current gradient.
Then we update the parameters themselves, which we call theta.
The update is just the old value of theta, plus the new momentum.
For this lecture, it's helpful to present momentum and a slightly different way.
Specifically, we now treat the momentum value as kind of the negative of what it was before.
The update for MFT is now the weighted some of the previous M of T minus one and the current gradient
G of T to update theta.
We now subtract them of T times the learning rate.
Ayda.
Now, since all these new symbols are just constants, you can show that the two methods of presenting
momentum are equivalent.
Let's also review arms prop arms prop also proceeds in two steps.
First, we update our cash, which is now renamed V.
Notice how similar this is to our modified version of momentum.
But instead of being the weighted some of the gradients, it's the weight, some of the gradient squared.
The second step is to update the parameter theta as usual, except instead of having the same learning
rate for all time, we divide by the square root of the cash.
OK, so for the next phase of this lecture, we're going to take a small digression and talk about moving
averages.
By doing this, we'll begin to make sense of why momentum in arms Propp take the forms that they do
to understand the moving average.
A nice place to begin is just the regular average.
Suppose I have a set of data points X one X two all the way up to the average of these data points is
exactly what you expect.
Sum up all the X's and divide by the total number of data points, which is T.
OK, so pretty simple so far.
The next step is to consider what if I have so much data that I cannot fit it all into memory at once?
For example, the data lives in a file that's one terabyte in size.
Unfortunately, your computer is not capable of storing one terabyte of memory.
Or consider another scenario where you have a robot interacting with the world.
In this case, each data point arrives at the robot's sensors at each instant in time.
I can't see any of the future data points, so my average can only consist of the past data points and
the current data point.
If I want the average of all the data points I have seen so far, let's assume that the current time
is t then sure I can add them all up and divide by T, but that seems wasteful.
You can see that as T increases.
The number of things I have to add together also increases.
Therefore, on each step my robot will have to do more and more computation in order to compute this
average.
We say that this computation is of T because the number of steps to perform the computation is proportional
to T.
My claim is that if we have a robot streaming in data one point at a time, we can actually make this
computation of one.
In other words, our computation is constant no matter how much data we've collected specifically that
we can compute the current average by somehow making use of the previous computations we've done, and
even more specifically, the previous average.
To understand how we can do this, let's manipulate the equation for the sample mean, firstly, notice
that our notation for the mean time T is exposed.
Subscript T, let's begin by splitting our sum by taking out only the final data point X of T.
The next step is to multiply out the one over T term.
The next step is to replace the sum from little T equals one up to Big T minus one specifically, we
can replace this with the sample mean at time T minus one.
As you recall, the sample mean a time at T minus one is just the sum divided by T minus one.
OK, so now what do we have?
We have exactly what we discussed previously.
We have an equation that allows us to compute the current average X party in constant time.
As you can see, it only involves something to terms the previous average X bar T minus one and the
current sample X of T.
The next step is to manipulate our expression even more specifically, let's take the phrase constant
T minus one over T and divide everything by T, then we get one minus one over T.
This is helpful because now that one over T term appears twice.
The next step is to ask the question, what if we replace one of our teeth with a constant let's call
it Alpha in this case, we are changing the answer, but we are doing so in a way that might be helpful
later on when this was equal to one over T, we saw that this gave us the regular sample mean.
As you can see, the function one of our T decreases as T increases.
However, it turns out that this will weigh all of the X's just right so that every X we have considered
so far is weighted by one over T note that we have just proven this and as you know, it leaves exactly
to the sample mean.
Importantly, this means that if we replace one over T with a constant, we will no longer have the
sample mean.
In fact, when we have a constant, this gives us the exponentially weighted moving average as an exercise.
You might want to try manipulating the equation for constant alpha and put it back into a summation
so that it depends only on the X's and not any previous averages.
What you should see is that the weights decay exponentially as you go further and further back in time.
This is opposed to the regular average where all the weights are equal.
Basically, this means that for the exponentially weighted moving average data points, which are more
recent or more highly weighted.
Now, just by convention, we can replace Alpha with the new symbol, specifically, let's introduce
Beta, which is equal to one minus Alpha.
We'll call this the decay.
Note that this doesn't change anything.
Only the symbols are different, but the functionality of the equation is still exactly the same.
At this point, you should recognize that this is exactly the same as the cash update for arms probe.
It is also the same as our modified momentum update.
So an arms prop, where do we use this exponentially weighted moving average?
In fact, we use it on the squared radians, so it's worth considering.
What is this the average of what are we estimating now trivially?
We are estimating the mean of the gradient squared.
So an arms probe, because we eventually end up using the square root of the cache.
Now, the name RMX prop makes sense.
Our math stands for Root Mean Square.
Specifically, we are estimating the square root of the mean of the square of the gradient.
However, we can go further than this.
What is the sample mean?
You may recall from statistics that the sample mean is used to estimate expected values.
Now, what is the expected value of the square of a random variable?
Well, it is related to the variance.
Specifically, the variance is equal to the expected value of the random variable squared, minus the
true means squared.
However, when we just have the expected value of the square, we often call this the second moment
in general, the expected value of X to the power N is called the enth moment.
In statistics, it's often the case that what we care about are the first and second moments, there
are some practical and theoretical reasons behind this, but they are outside the scope of this lecture.
Intuitively, we know that what we often care about when it comes to something random is the mean in
the variance.
We know that the second moment is something like a variance.
Similarly, the first moment is simply the mean.
That is the expected value of X to the one or simply the expected value of X is the mean.
At this point, we can recognize how this relates to momentum and arms, prop momentum essentially estimates
the first moment of the gradient using an exponential moving average.
Similarly, arms prop essentially estimates the second moment of the gradient using an exponential moving
average.
Now, since we might want to turn these two value separately, we will give each of them a possibly
different value of data.
We'll call the beta for the first moment beta one and we'll call the beta for the second moment.
Beta to the last step is to essentially put these two ideas together.
And that gives us Adam.
In order to make sense of how to put these two ideas together, let's remind ourselves what momentum
in arms prop look like when they are apart from momentum.
We subtract the learning rate and multiply by four arms prob.
We subtract the learning rate multiplied by the gradient and divide by the square root of V plus some
tiny value.
Thus for Atom we simply combine these two together.
That is to say, we subtract the learning rate, multiply it by M and divide by the square root of V,
plus the usual tiny value to avoid dividing by zero.
Now, this is the essence of the Adam optimizer, but note that there is still one more step before
we are done, since this lecture has been quite long so far, and this is a different topic, we will
save it for the next lecture.