Instructor: Hello and welcome to this tutorial.
All right, so in the previous tutorials we made the brain
or if you want, the brains for the A3C.
Now we need to train this brain,
but in order to train these brains,
we need an optimizer.
That's the tool we'll use in stochastic gradient descent
to update the weights according to how much they contribute
to the error between the predictions and the targets.
And what we did up until now in the first and second module,
we used the Adam Optimizer by Torch in the training.
But as I told you, we are dealing
with a very challenging problem that is breakout.
And the A3C algorithm by itself
is not enough to solve this problem.
We need some customized optimizers
and a lot of different tricks to solve this problem
without waiting for ages.
So, that's the purpose of doing this
and that is why we have a separate
custom optimizer based on the Adam Optimizer.
And that is contained in this shared Adam class.
And why shared Adam?
It's because it is actually the Adam Optimizer
but that will work on shared states.
So, we're going to explain how it works in this tutorial.
So we're gonna go through the different functions here
without coding them because, you know,
we want to keep some energy for the next implementation
that is the train of Python,
which will take more than 100 lines of code.
So be ready for that.
And therefore we will try to explain what's going
on here in one tutorial,
this tutorial.
And let's start right now.
All right, so first we introduce this class
Share Adam that will contain three functions.
The init function, the shared memory function,
and the step function.
So, what we do first
is that we inherit from
Optim.adam,
which is of course the Adam Optimizer
that we get from the Optim module
from the Torch library.
So here we apply inheritance to get the tools.
All related it to the Adam Optimizer.
And then we start with the init function.
So what happens here,
first we use a super function to inherit from all the tools
and all the basic parameters from the Optim.Adam class.
And these basic parameters are here
paras, learning rates, betas, epsilon,
and weight decay.
And then we start a full loop.
This first full loop for group in self.param groups.
So first, what is param groups?
Self.param groups contains all the attributes
of the optimizer.
And among these attributes we have the parameters
that we have to optimize.
These parameters that we want to optimize are the weights
of the network that are contained
in self.param groups
params.
So, there we go.
Group belongs to self.param groups.
And here we have this second full loop,
which will get these parameters that we want to optimize
and that are exactly contained in self.param groups params.
So basically,
we go through self.param groups
that contains old parameters.
And for each group of parameters and self.param groups,
we are gonna go through
the parameters that we want to optimize.
Therefore, for P in group params here means
for each tensor of weights that we want to optimize.
So for each tensor of weights that we want to optimize
and then what happens inside this loop
with these full lines of code.
Basically what happens is
that the update made by the Adam Optimizer
is based on an exponential moving average
of the gradient.
That's this line of code here.
That's the exponential moving average of the gradient
of moment one, that is of order one.
But the updates made by Adam is not only based on that
it is also based on an exponential moving average
of the square of the gradient.
That is an exponential moving average
of the gradient of moment two or order two.
So here is the exponential moving average of order one
and here is the exponential moving average
of order two.
For each of them, the EMA of the gradient.
So that what happens here.
And now, if you want to get more in depth
of how the exponential moving average works,
well I highly encourage you to have a look
at this research paper,
"Adam: A method for Stochastic Optimization"
because basically,
the Adam Optimizer that we're implementing right now
is based on the algorithm one here.
So if you want to have more details on
how the algorithm works,
well this paper will be definitely helpful.
And then you have some further explanations
on the algorithm with the Adams update rules.
And so you know that
only if you want to attack this before
attacking the big train function that we'll make afterwards.
Okay, so let's go back to Python.
And now let's move on to the second function,
share memory.
So now I'm just gonna say a few words.
The idea of this share memory function is kind of
like tensor.CUDA.
You know CUDA is an accelerator based on the GPU.
And so basically what happens here is that
we have these tensor of the states.share memory
here, here, and here
that behave a little bit like tensor.CUDA.
So you know, accelerated computations.
But the difference is
that here,
the tensor.share memory
send the computations to a part of the GPU
or the CPU
that is accessible to all the paralife thread.
So that's basically what is done here.
That's a little bit like tensor.CUDA,
but it's only sent
to a part of the GPU or CPU accessible to the
paralife thread.
All right, and then we have the last function step.
So you know this function,
it's like the step method of the Adam Optimizer
that we already used in this course.
And so again,
this is based on the algorithm one of the same paper
that we saw before.
So this algorithm,
so again, if you want to understand
in details the following lines of code,
well again I anchored you to have a look
at this algorithm one by this paper.
And besides what is done here is not totally compulsory
because this is actually a copy paste
of the step method of the optim.adam class.
So, basically what is done here
we could have done it by using our inheritance
because here we inherit from optim.adam.
And so to use our inheritance,
well, what we can do instead of doing all this
is just-
I'm gonna write here as comment,
is just use the super function
that we applied to our shared
Adam class,
then our object self.
And here we just add step with parenthesis.
Step is the method of the optim.adam class.
And that's exactly the same.
That's why I was just saying that here is just a copy paste
of the step method of the optim.adam class.
So I think that if you replace all this
by this super function apply to share Adam
and the step method,
well we might get exactly the same thing.
All right. So,
that was interesting to have a quick look at it.
Basically, you can see this as the Adam Optimizer.
It's like we had a deeper look at it.
But again,
if you want to go in more details of all this
and if you want to understand what happens behind the scene,
well I encourage you to have a look at this research paper.
I'll put the link in the comments here.
You know, remember you will have all the codes commented
in great details.
So it's really good if you can have a look at it.
And now I hope you have some great energy
because we are going to move on to the train file,
which will contain this huge train function.
And that will basically train our brains,
which now we can do because we have our optimizer.
So have a good break now.
Have a good sleep.
And whenever you feel in great shape,
let's move on to the next step.
Until then, enjoy AI.