Instructor: Hello and welcome to this tutorial,
almost the final tutorial of this module.
I'm just going to explain the main code
that will execute the whole thing
before we get the exciting results
and watch the videos.
So, this is the main code,
and as you can see, this is pretty short.
We start by importing the libraries and the modules,
and also the different classes and functions we made,
like active critic from our model file,
the train function from the train file,
and the test function from the test file,
and of course, we import our optimizer.
Then we start with the first section
where we get into a class,
all the parameters.
And this is this params,
remember this is the params object
that we created from this params class.
Each time we got a parameter
like the learning rate, gamma, or tau.
So let's go through them quickly.
This first one, LR here, is the learning rate.
So as you can see, we choose a small learning rate.
This second one is the gamma parameter.
Again, we take it as 0.99.
We take a one tau parameter,
a seed of one,
16 processes,
20 steps,
and max length of 10,000.
Remember we spoke about that.
This is the parameter we set
to make sure the agent doesn't get stuck indefinitely
into a state of the environment.
So this will stop the game
if the episode length
goes over this maximum episode length.
And eventually, of course,
we get the name of our environment,
Breakout V zero.
And by the way, you can also play
on some other environment
just by changing this name of the environment here.
So if you want to play to some other Breakout versions,
or even some other Atari games,
well you can simply replace this Breakout V zero here
by some other games.
But I can tell you that Breakout V zero
is already very challenging.
All right, so all the parameters here.
And then there is the main code for the main run.
And so here, let's see what we do.
In this first line, we set one thread per core.
Then in the second line, we get all our parameters
by, you know, creating an object of the params class,
which will get and initialize all these parameters here
because they are the variables attached
to this params object.
Then we set the seed,
then we get our environment
using the create atari env function
with the name of our environment,
which is Breakout zero.
You see cell dot env name,
and therefore params dot env name
is Breakout zero.
So that will get us the environment of breakout.
And by the way, this is not the usual way
of creating an environment,
but you know, to improve the whole process
and to improve the performance,
well we use this
to actually create an optimized environment.
And this, we do this thanks to Universe.
Universe is a package
that comes with all the packages you installed
on OpenAI Gym.
Well, thanks to Universe, we get an optimized environment.
This is what is all about here.
Then we get our shared model
by creating an object of the active critic class.
And so here it's important to understand
that this shared model
is the model shared by the different agents.
So we have different threads in different cores.
And speaking of threads,
at the next line, here,
shared model dot share memory.
What we do is we store the model
in the shared memory of the computer
so that all the threads can get access to it
even if they are in different cores.
So that's what we do here.
This is to enable this.
Then we get our optimizer
linked to the parameters of our shared model
and with a learning rate of 0.0001.
And again, it's important to understand
that the optimizer is also shared
because it's gonna act on the shared model.
And same, at the next line, optimizer dot share memory,
we store the optimizer in the shared memory
so that's all the agents can get access to it
to optimize the model.
Then we initialize our processes.
So the test process doesn't update the shared model,
but it just uses it to try it on one part
and print the score and record the videos.
So that's exactly what it's done here
with target equals test.
That's the test process.
And this process here
is got from torch dot multi pre-processing.
So here, and what it does is that it basically
runs a function on an independent thread.
So then when we do P start,
we start a new process,
which was the one initialized here at this line.
And then with this process dot append P,
we add the process in the list of the processes.
And finally, in this loop here,
we just do a loop to run all the other processes
that will be trained by updating the shared model.
And that's basically what happens
in the last lines of code here.
So, if you don't wanna get into the details of it
the important thing to understand is that
this will run the processes in an optimal way,
and therefore we should all be good to execute this code
and to have a train model and eventually watch the results.
So I can't wait to do that.
This is going to be pretty exciting.
I will try to find (indistinct) now
so that we can all watch it together.
And so until next time, enjoy AI.