-: Hello and welcome to this tutorial.
Now we have the brain of the model.
We also have the optimizer.
So basically we're ready to train our different agents.
That is our different brains.
So that's from now.
That's where will make this big train function
which will contain all the A three C algorithm
and therefore what we're about to implement in this train
that PyPI is just this huge train function.
There will just be this big train function.
Nothing else.
No class, but we will use this train function
and the last step of this module with domain code.
So there we go.
But before we start, you can notice
that we'll first we import some libraries.
So that's the classic libraries with the torch module
I mean the torch library.
Then the ends library to create the Atari environment,
which will be Breakout.
Then we will of course import our actor-critic class
from our model file.
This one.
And finally we will use variable from torch.autograd
to run highly performing computations of the gradient
thanks to the dynamic graphs.
And then we have this ensure shared grads function
which I didn't want to spend too much time on this
because well first this is just a function that will
make sure everything works correctly.
If the model used by the agent
doesn't have any shared gradient.
So that's why it's called ensure shared grad.
And the other reason is that
I don't think this function is necessary
but we never know, and at least with this
we'll be a 100% sure that the code will execute properly.
But that's not really important.
What we must focus on is this train function
that we'll start making right now.
So here we go.
Deaf
(Instructor typing)
and train.
We'll simply call it train.
And this train function will take several arguments.
The first one is rank.
I'm gonna explain what it is in the second.
The second one is params,
so that all the parameters of the environment.
Then the third parameter is going to be shared model.
So you know, the shared model is what the agent will get to
run its little exploration on a certain number of steps.
And then finally, the last argument is going to
be the optimizer that is the one we made earlier.
So perfect, that's our four arguments.
And now we are ready to start implementing this strain
function. So the first thing we'll do is, you know
you remember what A three C stands for.
It stands for asynchronous actor-critic agent.
So in A three C there is asynchronous.
So as you understood, we have to desynchronize
each training agent, and to desynchronize them
we're gonna use the rank to shift each seed with this rank.
So this rank parameter here, is just to shift the seed
so that each training agent is desynchronized.
So for example, if there is N training agents
then the ranks will go from one to N,
and there there will be one integer per agent from one to N.
So when we shift the seed by one thread
all episode random numbers created
by the thread will be totally independent
from the other threads.
However, the seeds are fixed numbers
so when we reproduce the experience
we will find exactly the same events.
And that's because it's deterministic
with respect to the seed.
So it's important to understand that
and that's why the first thing we need to do is to
desynchronize each training agent
by using the rank here to shift the seed with the rank.
So lets do this.
To do that, we're going to take our torch library.
Then we're gonna get the seed
with manual
(Instructor typing)
underscore seed parenthesis.
This is a function.
And now we're gonna take the seeds of all the agents
which we can access with params dot seed,
(Instructor typing)
and to shift them by the rank to desynchronize
each of these agents.
We will just adhere plus rank.
And that will shift the seed with the rank
to desynchronize each training agent
because there is one seed for each training agent.
All right, first thing done.
And now next step.
The next step is to get the environment.
So we're gonna create a new variable that we're
gonna call end.
(Instructor typing)
And now we will use to create Atari end function
from the end module to create the environment for breakouts.
That is to get the environment of breakouts.
So we take this function,
(Instructor typing)
create Atari end, and now we have to input just one argument
which are the parameters of the environment.
And we have them because this is one
of the input of the train function.
This is the params here
which will be the parameters of the environment of breakout.
And therefore to get the breakout environment
we take this params argument, then dot
and then we get end name, which in the future that is
in the next code with the main function that will execute.
The whole code will be Breakout, Breakout VZR.
All right?
So that gets us the environment.
Perfect. And now next step is to align the seed
of the environment under one of the agents.
And why do we do that?
It's because remember, each agent
of the A three C model has its own vision
of the environment, like its own copy of the environment.
And therefore we need to align each
of the agents on one specific version of the environment.
And to do that, we're gonna use the seed
because each seed determines a specific environment.
So by associating a different seed to each agent,
well we'll get exactly what we want.
That is that each agent will have its own environment.
And so how can we do that?
We can take our environment, then add
dot then use the seed function to,
you know choose the seed we want to get for the environment.
And so now to align the seed of the environment
to the seed of the agent.
Well we simply need to get this because this corresponds
to the seed of the agent that was shifted thanks to rank
to get desynchronized training agent
because they're all on the different seed.
(Instructor typing)
So we just need to paste that here.
And this will align the seed
of the environment under one of the agent.
(Instructor typing)
Okay?
Now we are gonna get our model that is our A three C brains.
And so that is now that we're gonna use
the actor-critic class from our model file.
So we're basically going to create an object
of this actor-critic class.
And we're gonna call this object model or brain if you like.
But basically this object will contain all the convolutions
the LSDM, the linear full connection
and the forward function to propagate the signal.
So it will basically contain the brains of the actor
and the critic with the ability to propagate the signal
throughout the brain to get the final output.
So lets do this.
Lets create our model.
So as we said, we want to call this object model.
And so we create an object of the actor-critic class
and therefore we take our class
(Instructor typing)
actor-critic and now remember what arguments
we need to input.
That's actually the arguments of the init function.
So self, we don't have to input it, you know
that's what we have to do to use the object
in the init method.
But then the arguments we have to input are numb inputs,
which is input shape, that is the dimensions
of our input images and the actions space that contains
you know, the set of actions.
So let's input these arguments in the train function.
So the first one, we can get it with our environment
and dot and then we use observation space.
(Instructor typing)
So that's the space of observations.
Then dot.
And then to get the number of inputs,
we get shape bracket zero.
All right, so that's for numb inputs.
And now for action space.
Well that's almost the same which we need to get it
from our environment that we already imported.
Then dot and then action space.
(Instructor typing)
All right and that gives us the argument
we need to input when creating an object,
the model of the actor-critic class.
Okay, so now we have our model.
And now the next step is to prepare our input states.
So remember we're still doing deeper reinforcement learning.
So the input states are the input images
and therefore this will be originally a MPIRE Ray
which will contain one channel
because we will work with black
and white images and it'll have dimensions of 42 by 42.
But it's important to understand and to keep in mind here
that the input states are the input images.
So first what we have to do is to get the MPIRE Ray
then we will convert it into a torch sensor.
But the first step as what we did previously
is to get an MPIRE Ray.
And to get it.
It's actually quite simple.
While first we need to create a variable
for the input state, which we will call state.
And this, to get the MPIRE Ray
we simply need to take our environment
and then add dot and then use the reset function.
And this will initialize state
as a MPIRE Ray of dimensions, one by 42 by 42.
One means one channel, so black and white image.
And 42 by 42 is of course the dimensions of the image.
The number of pixels on the widths
and the number of pixels on the height.
So basically that's just the dimensions
and that's the ones we'll work with.
And now, now that we have this actually MPIRE Ray
because this will get us these images of such dimensions
in MPIRE Rays, now we can convert them into torch tensors.
And to do this while we're going to update state again
because we don't need to keep the MPIRE Rays
and that's where we use torch, the torch module.
And remember we already did that for Doom.
We used the function from underscore NumPy parenthesis.
And inside this function, we need to input the MPIRE Ray
to which we want to convert into a torch tenser.
And that is the state.
The previous version of the state in MPIRE Ray will become
by applying the from NumPy function, a torch tensor.
So that just creates a tensor from the state.
And now we just need to initialize the done variable.
Remember the done variable is generally the variable
that says if an episode is over or if the game is over.
While here, we just want to introduce this done variable
and initialize it to true to specify
that this done variable will be equal
to true when the game is done.
So that will be useful for later so
that the AI doesn't play indefinitely to break out.
All right, so that was basically the beginning
of this train function with some initialization
and some things that we have to do.
The most important part here was
that we have to desynchronized each training agent.
So that's one first principle of the A three C model,
we have to apply.
And now in the next tutorial, we will proceed to
desynchronization with the shared model.
Let's not forget that there is the different models
but also the shared model which is a model
that all the agents share.
And so we have to synchronize with this shared model
so that each agent can get this shared model to proceed
to a small exploration of a certain number of steps.
So that's what we'll do in the next tutorial.
And until then, enjoy AI.