Narrator: Hello and welcome to part two,
training the AI with deep convolutional Q-learning.
That's right, now that we built the AI
with the architecture of the neural network,
the body, the way the actions are played and everything
it's time to train this AI with deep convolution Q-learning.
So that's from now that we will implement experience replay,
working with the Q values, working with the rewards
and there's even gonna be a bonus
which will improve a lot, the training process.
And that is called eligibility trace.
Eligibility trace is a powerful technique
which consists of accumulating the reward
over several steps and the Q values are learned
on this accumulation of rewards.
As opposed to before, where the Q values were learned
after each transition.
Therefore, after getting each reward.
This time we will be learning the Q values
after getting several rewards instead of just one reward.
So instead of having one transition after the other
and updating the Q value each time.
Well, the Q values are gonna be updated every n-steps,
because eligibility trace is rather called,
n-steps eligibility trace.
And N is this number after which
the Q values are gonna be updated.
And in our model here we're gonna have an equals 10.
So that means that will be a 10 steps eligibility trace
and therefore we will update and learn the Q values
every 10 steps after accumulating the rewards
on these 10 steps.
So that's a bonus
that will make our model even more powerful
and you will see that in the end
we will get outstanding results.
I was really amazed when I saw the final results.
I used to work on models that took a lot of time to execute.
The AI took a lot of time to train
but you will see that with this one
plus the neural network that we made, that is our brain
and our body here with Steph Max,
we will get a very powerful model
and therefore very powerful AI,
because you will see that it will ridiculise Doom.
You all understand what I'm talking about.
So as you can see in this part two
we are starting by getting the Doom environment,
and I actually prepare the lines of code for you.
We are just using the image pre-processing external file
from our working directory folder.
So basically the order is rather to first
take this line of code, gym.meg,
P packet Doom cord of V-zero.
So Doom cord of V-zero is the name of the environment
of the game we're playing.
So first we import the environment with this gym.meg,
that's what you can find on the OpenAI gym tutorials.
But then, we use this pre-process image class,
which is a class from image pre-processing
to pre-process the images
that will come into the neural network.
And we pre-process them so that they have a square format
with the dimensions 80 by 80, and that remembers because
in our neural network--
Well, we set our input images to have the dimensions
one by 80 by 80.
Remember one is the number of channels.
And so, one means that we're working with
black and white images.
So that's the gray scale here.
And 80 by 80 means that the dimensions of our input images
will be 80 by 80.
And that is what we set in the neural network,
but of course then we need to specify this
when inputting the images, which is exactly what we do here
with this pre-processed image class.
And then, after we import the environment
with the right format of the input images,
while we import the whole game with the videos
with this line of code.
And remember, the cool thing about this is that
in the end we'll see the videos of our AI playing Doom.
So we will see how it will kill the monsters,
try to reach the best and everything.
So that will be super exciting and remember that
these videos will go into this videos folder.
All right, and last line here, but I want to show it to you
because that's important, that's now more related to
the AI that we're building.
Well, remember that our neural network
takes as input, number actions.
That's because we want to make an AI that we can test easily
on several environments, on several Doom environments.
And since the different Doom environments
have different number of actions,
well, we specified this number actions variable
as the input of the cnn, the brain.
And therefore, now what we're gonna do is
get this number actions variable using
the Doom environment that we just imported
and create it into this variable.
And later this number actions variable
that we're about to create will be the input of the brain.
So let's do this, I'm introducing this,
real now, variable number actions.
So number actions equals,
now we're gonna take our Doom environment,
that is the variable that we created.
So Doom environment, then we add here dot
and then, well here we go,
we take the first here, action-space.
That's the set of your actions.
I encourage you to have a look at the OpenAI tutorials
to see how it works, to understand how
the OpenAI gym environments work.
But basically, this is the set of actions.
And from this set of actions, we can access
the number of actions in the environment.
And to do this, we add a dot and n.
N is the number of actions.
And therefore Doom-n-dot-action-space-dot-n,
will return seven.
It will return seven because there are seven actions.
I know that we can see six actions
in the Doom environments on the OpenAI gym page,
but I think we can also run.
And so you know, we can move forward, move left,
move right turn left, turn right and shoot.
And besides, we can run.
So that makes seven actions.
All right, and that's it for getting the Doom environment.
We have the Doom environment, we have the number of actions.
So we have so far everything that we need for our brain.
We will then just create an object, a brain object,
which we'll call, cnn, in minimal letters.
And since the init function takes number of actions
as argument, well we will input the number of actions
in the cnn object that we will create.
And then of course we will create the body
and eventually the AI.
And that's why the next section, I'm gonna call it,
building an AI, because now
we can build as many AI as we want.
That's the awesome thing about object oriented programming,
we can build any AI as we want.
And so we're gonna build our AI
that has this sophisticated brain
and that's exactly what we'll do in the next tutorial.
Until then, enjoy AI.