In this lecture, we'll learn about one more step to finalize our atom optimizer equations to understand
why we need to do this, we have to understand what the exponentially weighted moving average can be
used for.
One way to look at it is that it's just another way of computing the average.
Specifically, it's useful when our data is non stationary.
That is, the true expected value might be changing over time.
Therefore, more recent values are more useful than older values.
Another way to look at the exponential moving average is this.
Suppose that you have some time to read signal such as a stock price.
Clearly, this is a non stationary signal, so it's a good candidate for using this kind of average.
Well, what happens if we apply our moving average to this time sorry signal.
Let's call the input time series of T and let's call the output time series Y of T.
Well, what you get is you get a smoothed version of the input signal, that is, it generally follows
the same trend, but in a much smoother manner without lots of random jumps up and down.
This is useful in finance, for example, when you want to know the trend of a stock price, but you
don't care about minor fluctuations.
If you have experience in signal processing, you might know this by another name.
This is called the low pass filter.
The reasoning behind that is this slow fluctuations have low frequencies, whereas fast fluctuations
have high frequencies.
Generally speaking, the low frequency movements are what we actually care about because they're usually
larger in magnitude.
The high frequency movements are usually very tiny and tend not to be that significant.
So by using a low pass filter, we are essentially passing our input signal into an algorithm that removes
the high frequencies we do not care about.
There is one small problem with our low pass filter.
Note that our output we have t always depends on the previous value Y of T minus one.
But if we are trying to generate the first output Y of one, then what is Y of zero?
A convention is to simply set this value to zero.
But what happens when we do this?
The answer is that at the beginning of our output, all the values are biased toward zero.
Now, if you never implemented this before or you've never taken one of my courses where you got an
opportunity to implement this, it would be very useful to try and implement it yourself, to see what
I'm talking about first hand.
It should be a pretty simple exercise and you can try this on any time series you choose, such as the
price of your favorite stock.
Now, what does it mean that the beginning values are biased toward zero?
Well, this is clearly not good because it takes our filter some time to catch up and to start producing
useful outputs as a side note in other fields.
There are simple solutions to this, such as simply setting Y of one to be equal to X of one instead
of zero.
However, in deep learning we employ a method known as bias correction.
So what is bias correction, bias correction means that instead of using what we would usually use Y
of T, we adjust this value by some factor to get Y hat of T specifically, we divide by one minus beta
to the party.
One way to see that this works is by noting that since Beta is a number less than one beta to the power
T will approach zero as he gets large.
Therefore, when T is large, we essentially divide by a number close to one and we have T is very close
to Y of T, which makes sense.
This is because bias correction is only really needed when he is small.
So what happens when he is small?
The simplest way to see this is to plug some numbers in, let's suppose that Beita is equal to zero
point nine nine.
Then why of one is equal to better times.
Why of zero plus one minus better times X of one since Y of zero is zero, this is just zero point zero
one times X of one.
Clearly this is not good since we are only taking one percent of the true input X of one.
However, what do we get for Y height of one?
Well, we take Y of one and divide it by one minus beta to the power one.
Of course that's just one minus beta, but one minus beta is just zero point zero one as we have just
seen.
Therefore Y had of one is just zero point zero one at times X of one divided by zero point zero one,
which is just X of one itself.
Thus Y have one is no longer biased.
It's exactly equal to X of one.
How about when C is equal to two?
In this case, Y of two is equal to better times.
Y of one plus one minus better times X of two, which is equal to zero point nine nine times Y of one
plus zero point zero one at times X of two.
However, we know that we have one as equal to zero point zero one at times because of one, so we can
plug that in.
We then get why of two sequel to zero point zero zero nine nine times out of one plus zero point zero
one a time XXXIV two.
So now we basically have about one percent of acts of one and about one percent of X of two.
Still not a good estimate for Y of two.
But what happens when we do bias correction?
We take one of two and divide that by one minus beta squared.
Now the numbers get a little messy, so you'll have to do the calculations on your own at home.
But what you should end up with is this.
You should get why had to zero two zero point four nine seven times X of one plus zero point five zero
three times X of two.
So does this make sense?
This makes perfect sense.
We take about half of X of one and about half of X of two.
We take a little more from X of two since X of two is more recent than X of one.
Furthermore, notice how these adjusted weights always add up to one, which makes a lot of sense.
OK, so how can we incorporate this into our atom optimization?
Well, we know that we are doing two exponential moving averages, one for M and one for V.
Therefore, we simply replace what we had before with these byas corrected versions of Maanvi.
So now, instead of subtracting the learning rate, multiplying by M and dividing by the square root
of V, we simply replace MLV with their bias corrected versions.
Mhat and we have.
In total, our final algorithm is this before we start a training loop, we initialize M zero to zero
zero two zero and the time index T to zero.
Then we answer our training loop inside the loop.
We increment the time index by one.
Next we find MFT using the usual recursive equation with the hyper parameter decay rate beta one.
Next, we do the same thing for VTE using beta two.
Next, we perform bias correction to get Mahadev T and we have T.
Finally we perform our gradient update using the usual equation.
Note again that this is for some generic parameter vector theta.
In practice this represents a collection of all the parameters of your model.
So you will need EMS and Vee's of the same size as all of your parameter vectors matrices and Tensas.
The final thing I want to discuss in this lecture is what values do we choose for these type of parameters?
Previously, I told you that Adam optimization is very robust.
That is the same values work for a wide range of problems with practice and experience.
You may find that this is the case with Tensor Flow and PI torch.
Note that these default values are built into the optimizers, so you rarely have to think about them.
However, if you do ever end up implementing Adam from scratch, here are the suggested default values
for the learning rate.
A typical default value is ten to the minus three.
For Beta one, we use zero point nine.
As you recall, this is a typical value.
When we are using momentum for Beta two, we use zero point nine nine nine.
As you recall, this is a typical value.
When we are using arms prop for Epsilon, a typical default value is ten to the minus eight.
As a final thought to conclude this lecture, I want to mention that while Adam is a common default
choice, you will still encounter scenarios where it is not the best choice.
In fact, sometime after many people had adopted Adam as their go to default choice of optimizer, a
paper was written that showed a regular stochastic gradient descent with momentum performs better than
Adam.
This paper is called the Marginal Value of Adaptive Gradient Methods in Machine Learning.
Personally, before the invention of Adam and even after the invention of Adam, SAGD with momentum
was often my own default choice.
This reminds me of a rule that I often repeat to my students.
This rule is that machine learning is experimentation, not philosophy.
Oftentimes students just want to be told to one way of doing things so that they have less choices to
make by themselves, which I suppose is more comfortable.
But the reality is, unless you try it yourself, you will never know what the real answer is to put
this in a different way.
Ask yourself this question.
What should you do when you want to know the output of a computer program, specifically a computer
program where you train your model on some data set using a variety of optimizers?
Well, it would be silly and inefficient to ask me or anyone else what the output of the program will
be.
Similarly, it would be incorrect to try and guess what the output of the program will be instead,
if you want to know the output of a computer program.
Let me tell you an ancient secret that only true goosh know the answer is to simply run the computer
program and look at the output yourself.
As always, I am glad to be the source of many surprising and unusual facts.
Thanks for listening and I'll see you in the next lecture.