Hadelin: Are you ready?
Let's do this.
Let's start by installing
all the system dependencies for ViZDoom.
Let's click this play button
and now it will install all the dependencies,
as you can see, like Pillow or SciPy.
And also some other dependencies
that are gonna be needed to run this successfully.
But all the rest, like PyTorch,
the gym modules are already installed.
So that's really awesome.
That's really the beauty of Google Colab.
And that is thanks to which,
none of you will have any issue,
executing the code and visualizing the final result.
All right, so this is gonna take
actually one or two minutes,
so I'm just going to fast forward here
and I'll see you very soon
for the rest of the execution of the code.
(smacking lips) All right, we seem to be at the end
of the installation of the system dependencies.
As you can see, it is downloading them,
installing them,
while connecting them with all the right requirements.
And in a matter of seconds it should be done.
Right.
And let's see,
three, two,
there we go.
Successfully installed everything.
Don't worry about these errors here,
they won't impact the execution of the code.
But everything is successfully installed.
Just as we want, all right?
So, now next step, very important,
look at this important note.
It says that after installing all dependencies
basically after executing this first cell here,
you have to restart your Runtime.
Otherwise you will get some execution errors here.
So let's do that quickly, it's very simple.
You just need to click Runtime here,
and then restart Runtime,
and then yes, all right.
This will restart your Runtime.
And now you can just execute all these cells
by just clicking the play button.
So let's do this.
Starting with this first file, image pre-processing.
All right, so let's first import the libraries,
then pre-process the images
with the pre-process image class.
And now we already moved
to the experience replay implementation,
meaning this one.
All right.
So let's do this.
We first import the libraries,
then we define one step of the environment,
then we make the AI progress on several end step steps
with the end step progress class
exactly the same as what we have in the folder.
And then we implement the experience replay
by building this replay memory class.
All right, all good?
And now we move on to the third file.
This one, AI for Doom, ai.py.
And we first import the libraries,
then, all right so it takes a little time here,
because we import all the torch modules,
then we import the packages for OpenAI and Doom.
So with the gym and ViZDoom gym and the wrappers.
Okay, so all good.
And then we move on to part one here,
building the AI.
Where we're gonna make the brain, then the body,
and then assembling everything.
So let's first make the brain,
all right,
with the CNN class.
Then, let's make
the body,
with the Softmax body class.
And then let's make the AI,
with the AI class.
All right?
So all good.
And now we already move on to part two
where we're gonna train the AI
with of course, Deep Convolutional Q-Learning.
All right, so exactly the same as what we have here.
We first get the Doom environment with our new modules,
ViZDoom, don't worry about this,
this is not an error.
Then we are gonna build the AI by, you know,
creating the different objects, the brain, CNN,
the bodies of Softmax body
and the whole AI containing the brain and the body.
All right, did I execute this?
Yes.
Then, we set up experience replay
with end steps and the memory as an objective replay memory.
And then we implement eligibility trace,
all right, to improve the performance.
Then we make the moving average
on 100 steps with the MA class.
And finally, my friends, are you ready?
Well this is now time for, you know,
the very exciting part
where we're gonna train the AI over 20 epochs.
All right so you will see
that this will be already a bit long.
You know, it will take like,
maybe one or two hours because I increase dimensions.
Feel free to reduce dimensions back to 80 by 80
if you find this too long.
But trust me,
you will have much better videos with these dimensions.
Okay?
So.
Are you ready?
Let's do this in three,
two,
one,
go.
All right, so this will execute the code of the training
and in a few seconds we should be able to see
the first epoch,
which will have, of course, a negative reward.
But you will see that, you know, over the epoch's
that reward will increase little by little,
until reaching positive rewards
and on then until reaching hundreds of reward.
Well, let's aim for that.
Actually, let's hope that with 20 epochs
we will have some final rewards at some hundreds,
you know, like 100 or 200 or 300.
Because with these rewards, I experimented with them,
actually, yes, first epoch, negative rewards minus 98.
So I was saying that with rewards at around 100, 200, 300,
we will get already some great result.
You know, we will see the AI managing to
either kill some monsters or avoiding them
or you know, moving towards the vest.
Okay?
So, that's the first epoch.
Epoch 1 minus 98.
And then you know,
we'll see epoch number two,
with maybe already a better reward.
But you know, at the beginning, of course,
the AI is not trained.
It is exploring the environment, right.
Remember this trade off in reinforcement learning,
exploration versus exploitation.
Well, at the beginning the AI is purely exploring,
and then it's gonna train
and then it's gonna become smarter and smarter.
And that's when it's going to reach some high rewards.
So, it's totally fine
to have negative rewards at the beginning.
Maybe we'll get that over the three first epochs.
But then you will see that after I put number four,
five or six, well we will start reaching
maybe positive rewards,
and then hopefully rewards at around 100, 200 or 300, okay?
So, it's gonna take a little while as I said.
So we're not gonna stay here for two hours,
otherwise I will run out of things to say.
So, what I'm gonna do is,
I'm going to put a fun or cool music now, and there we go.
Epoch number two, minus 62.
So there is already some improvement, that's good.
But you will see that there will be
some even better improvements,
the more we progress in the epochs.
Okay?
So what was I saying?
Yes, I'm gonna put some cool music now
and play the training in accelerated mode.
And of course,
I'll see you at the end of the training
to see the final results.
All right?
So there we go.
three,
two,
one,
go.
(upbeat techno music)
(music volume increasing)
(Hadelin claps hands) (music stops)
All right, and here we are at the end of the training.
Congratulations.
You trained a Deep Convolutional Q-Learning model
on a very challenging application,
which is to play the game of "Doom".
So, what to say first?
Well, as we hoped for,
we reached some average reward at more than 100.
Then, what is important to say is, of course,
that with more epochs you will get higher rewards.
So if you're ready to, for example,
train this model for more epochs,
like 100 epochs, or even more,
in order to reach rewards
more about 300, 400, 500, or even 1000,
well, feel free to do it.
For example, you can let this run
for the night while you sleep.
And when you wake up in the morning,
you get your better results.
Note that you can also use in the Runtime at GPU, right?
If you change the Runtime type,
which I shouldn't do because otherwise
it will restart the notebook,
but, in the hardware accelerator here
you can choose GPU or even TPU.
But that's only if you wanna, you know,
optimize the performance and do some super hard training.
But here I just used the classic thing
because I just wanna show you how to execute all this.
And there we go.
Now we're going to execute the rest of the workbook
with this extra code,
only specific to this Colab notebook,
where, of course,
we're gonna visualize the AI in action.
All right.
So let's do this.
Let's first import the libraries, right?
All good.
Then, we're gonna print the input shape
and the number of possible actions.
All right, so here,
we're gonna get indeed that we have seven possible actions,
you know in the Doom Corridor environment.
So these are move forward, move backward,
go left, go right,
shoot,
and then maybe protect yourself or whatever.
I don't know what the last action is,
but something like that.
Then these are the dimensions of the input frame.
This corresponds to the height of the frame.
It has 240 pixels height.
This corresponds to the width of the frame, right?
It is 320
large.
And this corresponds to, you know,
the fact that we work with colored images
and the three here corresponds
to the three elements of RGB channels.
All right.
Then, let's execute the next cell.
Displaying a frame of the environment
just to see how it is like,
and indeed remember that's useful
to see the environment you're working with, right?
So here we see that we are in Doom Corridor.
But if you want to experiment
with some more environments here,
actually, you know, remember,
they are also in the main page.
Here, right here.
Yes you have all the lists at the environments, you know,
that I recommend experimenting with.
So, you know, if you want to have a look at another one,
well, you know I can show you actually,
let's get this one.
And then I'll go back to ViZDoom Corridor.
Right, so if I replace that by this one,
(keyboard keys clicking) right.
And I re-execute this, well,
we will get three actions this time.
Of course it is a more simple one.
And if we execute this cell,
well, we'll see this environment, right?
So it's just a way to quickly see
what you're working with and, and to get a preview
basically of what the AI will do in the environment.
(lips smacking) All right,
so let's go back to ViZDoom Corridor.
Let's re-execute this,
you know in case we need it for the next cells.
All right, perfect Doom Corridor.
Now let's move on to the final cell.
So this is a helper function
that will be used for the visualization.
So let's execute this cell.
And now, let's run the AI on one episode.
And here you will be able to understand the code
because you know, basically,
it is the process of, you know,
running the AI on a full episode,
where at each step, it is in a specific state,
it's going to play in action within the state,
then it's going to get the reward
and then reach the next state, right?
So this is the classic MDP process, right,
Mark of Decision Process.
And so there we go.
That's done for one episode.
And finally,
we're gonna get the video
of the game play of our AI,
who was trained for 20 epochs,
was able to reach more than 100 reward,
which will get us some pretty good results,
but maybe not reaching the vest, but it's okay,
you'll experiment that by yourself.
And now I would like you to click this folder button here
because I will want to show you how the video is populated.
So these are some folders containing, you know,
some elements like the frames.
You know, the frames of the results or some JSON files.
But really what we'll be interested in is the final video.
And to get it, we just need to click the play button here
and you will see that it will be populated
in the main folder here.
Don't miss it.
And it's okay, you know, it looks like the cell
was already executed, which is the case,
but you'll see that in a few seconds
we will see an AVI file,
which is a video file,
being populated here.
It will appear in like 10 seconds.
I can even do a, a countdown if you want.
So let's do this.
And 10,
nine,
eight,
seven,
six,
five,
four,
three,
two,
one,
go.
And now it should really appear in a few seconds.
Yeah, there we go.
All right, so agentgameplay.avi, that's your video.
So let's download it.
And it will be downloaded on your computer,
which I will find right here.
Right, so that's the video, agentgameplay.avi.
Make sure to open it with a video player
that has codex like VLC, right?
It won't work with QuickTime Player if you're on a Mac.
But it will definitely work with VLC.
So let's have a look.
And here is the video.
All right so, let me just press pause here.
All right, so here is the video, and as you can see,
so this is the 256 by 256 dimensions of the frame.
So as you can see, it's not very large, right?
So that's why I really wanted to work
with these dimensions instead of 80 by 80.
So now let's have a look at the video.
So this is the AI playing "Doom" on one episode.
And it has to avoid the monsters, not be killed.
It has to move forward to reach the vest.
So let's see how it does.
three,
two, one,
go.
(mouse button clicking)
All right, so first it gets shot.
It moves forward, okay,
and then it got killed by the monsters.
But, that's still quite good you know,
it understood that it had to move forward towards the vest.
Because the highest rewards is got by reaching the vest.
And of course, you know, if you really wanna see the video
of your AI reaching the vest,
and winning at this game basically,
you will have to train your AI for more epochs.
And maybe do some even other kind of improvement,
like tuning your brain of the AI,
or doing some parameter tuning,
like tuning the learning rate.
Well, you have many options.
If any of you get an amazing video,
or you know, the video of an AI
having reached a reward of more than 1000 for example,
well, feel free to share it in the Q and A.
I'm sure other students will be super happy to get it.
All right so, I hoped you liked implementing
the Deep Convolutional Q-Learning model.
Now we're gonna move on to the next part,
which will be about implementing the A3C model,
an even better and more powerful model,
which will implement to play the game of "Breakout".
So I'll see you in the next part,
and until then, enjoy AI.