-: Hello, and welcome to this tutorial.
I'm super excited
because we're about to make the A3C brain
that is the brain of our AI.
And speaking about brains,
I would like to highlight something.
Remember in the first module
we made a simple brain with only fully connected layers.
Then in the second module for Doom,
we made a brain that not only had fully connected layers,
but also eyes because we added the connected layers
which gave eyes to the AI
because it could observe the images
and understand what's going on inside.
And now we're gonna take it even at a high level
because we are gonna make a brain
that not only will have eyes and fully connected layers,
but also memory.
Because as I said in the previous tutorial,
we're gonna add a record renewal network
inside this big brain,
and that will give a long memory to our brain
so that it can understand the temporal relationships,
the temporal properties of the input images.
So there we go,
and an even more powerful brain.
I can tell you that the model
we're about to implement right now
is really, really powerful.
And we can see how building AIs
and doing deep learning,
doing deeper enforcement learning
is all about getting closer and closer
to how human brain works.
We started with the basic relationships that the brain
with the linear full connections,
then we added eyes,
then we added the memory.
Who knows what we're gonna add in the future models?
In 2018, maybe they will add
something that will make the brain look like
even more like a human brain.
But already with fully connected layers, eyes, and a memory,
we have already a really good and functional brain.
So let's do it.
Let's make this brain.
So as usual, we're gonna make a class for that
because it's gonna have a lot of properties
with the convolutions and the LCMs.
So we're gonna make a init function
to initialize all this,
create all these connections,
and then of course
we'll have the forward function
that will of course propagate the signal inside the brain
so that we can get eventually the outputs.
All right, are you ready?
Let's do this.
So class, we introduce a new class
which we call actor-critic,
because of course I'm talking about brains here.
But let's not forget that we're making the A3C model
which is based on the actor-critic principle
with separately the actor and the critic.
So we will make actually
one linear full connection for the actor
and one linear full connection for the critic.
You'll see how we will do that.
It'll be actually quite simple.
So actor-critic
and this actor-critic class is going to inherit
from the nn. module
so that we can use all the PyTorch tools.
So let's do this to inherit from the nn. module
while we need to take first the torch library
then dot then nn, and dot and module.
All right.
So that way we inherit from it.
All right.
So there we go with our first function
which will be of course the init function.
So we start with init,
init double underscore.
Then this init function is going to take
as argument self for the object.
Then the input shape that is the dimensions
of our input images, and we call it numb input.
And the action space
which is basically the space that contains all the actions.
But also, you know, from this action space
we can get the number of actions.
That is a number of possible actions
which we will actually get very soon.
So that's why we also need it.
So that's for the arguments, that's all we need.
And then let's go inside the function
and let's create all the variables proper to our brain.
But before we do that
remember what we have to do to activate
in some way the inheritance that we can use all the tools
from the nn. module,
we have to use the super function this way
inside of which we input actor-critic, that is our class
and then, comma, self for the object.
All right?
Then dot.
And there we go again with the init function,
there we go.
That gives us all the tools that we'll need
from torch to build our brain.
All right?
Then, well, it's time to make the eyes of the AI
that is the convolutions.
So we're gonna do it very quickly
because we already explained this
in details for Doom, because remember
the AI for Doom had eyes, so it's exactly the same.
We're gonna make some convolutions
and we will use a very simple architecture
with 32 feature detectors of size, three by three
a stride of two, and a padding of one.
So that's a pretty classic architecture
but that will actually be enough to, you know
make sure that the AI understands
what's going on in the breakout game.
All right, so let's make those convolutions.
So we start with self
because the convolutions will be variables of the object.
So self.com, we can call it com
and there's gonna be four convolutions.
So I'm calling this one cons one.
And there we go.
We take the nn module dot
and then we take the com 2 D class
because actually cons one will be an object of this class.
And then
inside first we input the input shape of the images.
So that's exactly what we have here.
So we can copy this and enter it as the first input.
Then the second argument is the number
of feature detectors or also the number of kernels.
So we're gonna take 32, as we just said, classic choice.
Then we need to choose a size of the kernel.
That is the number of cells
that will slide over the input image.
And so remember, we can either take
3, 4, 5, that's common choices.
And here we're gonna choose three.
And then we're gonna choose a stride of two
and a padding of one.
There we go.
So that's for the first convolution
that goes from the input image
to the first convolutional layers
composed of 32 convoluted images.
So now we're ready to make the second convolution.
So it's actually going to be almost the same.
So I'm copying this line and pasting that below
but pasting it below again and pasting it one last time
because we're gonna have four convolutions
with almost nothing to change.
So we can already replace here cons one
by cons two; cons one by cons three;
and cons one by cons four.
That will be our four convolutions.
And now of course, we need to change some things here
but not much because we're gonna keep a stride
of two for each and a padding of one
they will all have 32 feature detectors.
That is 32 output convoluted images.
But then here remember, this corresponds
to the left part of the convolution.
So actually that corresponds to what was
at the right part of the previous convolution.
You know, remember, it's like a domino,
so it's really easy.
And therefore here we have to input 32
and here as well. We're gonna see very easily 32 and 32.
All right, so to sum up, we start
with our input images that has numb inputs dimensions,
with the first convolution, we get 32 convoluted images
each one detecting a specific feature.
Then from these 32 convoluted images, we apply
the second convolution to get 32 new convoluted images.
Then same from these 32 new convoluted images
we apply the third convolution to get
32 new convoluted images again.
And then eventually from these 32 convoluted images
we apply the fourth convolution to get features.
All right, And this will be enough,
with this our AI will have a supervision,
it will detect the ball very well.
All right, so that's it for the convolution.
So that's it for the eyes.
And now let's take care of the memory.
This new feature of this brain we're implementing
as opposed to before with Doom,
not only it will have a supervision
but also it will have a super memory,
a long memory, because we are going to
implement an NSDM, long short-term memory,
which is this kind of record renewal network
that gives to your model some kind of a long memory
so that it can learn some long temporal relationships
from the past.
So same, we're going to create a new variable.
So I'm starting with self, and this new variable
we're gonna call it simply LSTM
because this will correspond
to the LSTM network inside the brain.
So LSTM, and now, before we write the code for the LSTM,
let's make sure we understand what this LSTM
part of the brain will do.
So as we understood, this LSTM is used
to learn the temporal properties
of the input, of the input images.
So for example, if the ball hits a break
the LSTM will encode the bounce.
So that's the first thing to understand.
It will kind of encode what's happening in the game.
Then the next important thing to understand when
we implement an LSTM is that we need to choose an order
of the temporal dependencies.
And here, since we're gonna feed our neural network
with a sequence of four images, then that means
that we can already learn some temporal dependencies
of order four.
That is some temporal dependencies,
where what happens at time T plus one,
depends on what happens at time T
T minus one, T minus two, and T minus three.
So that we can definitely do that.
But the good news is that we're gonna use an LSTM
and therefore we will be able to learn some
even more complex temporal relationships.
That is, for example, we can learn
some temporal properties where what happens
at time T plus one will depend on what happens at time T
T minus one, T minus two, T minus three down to T minus N.
And that's the long part in the LSTM,
long short-term memory.
With this LSTM
we can learn some very complex temporal relationships.
All right, so let's add our LSTM.
So to do this, we're gonna use the nn module
and then we're gonna add the Class LSTM cell,
which will create this LSTM object,
which will represent the LSTM part of the neural network.
Because right now, what's also important to understand
is that we're making a C RNN, you know
a convolutional recurrent renewal network
and the RNN part comes after the CNN part.
And therefore, right now what we need to input
in this LSTM cell
is first the size of the output after the convolution.
So that is 32 times three times three.
So this 32 times three times three is actually the output
after the four convolutions here.
But that becomes the input of the RNN, the LSTM network.
And now why does the output of the four convolutions
have the size 32 times three times three?
Well, don't worry, it's not that direct.
It's actually not a simple formula
but there is a formula to compute this number
of output neurons after flattening the pooled
and convoluted images of the convolutions.
But if we gather the terms of this big formula,
well, we get 32 times three times three.
I didn't wanna spend too much time
on this because we have a lot to do more.
And besides we already made a function
to compute this number.
Remember, it was for Doom
when we made this count neurons function.
So you can reuse it if you want, if you're not convinced
but that is just what we get after gathering the terms
of this big formula computing the number of outputs.
So that's for the first argument.
And then the second argument is going to be the number
of output neurons of the LSTM, and we're gonna go for 256.
Okay? And so what does that mean now?
That means that now we have a vector
that encodes each event of the game.
Or in other words, we have an encoded state.
And so that is now that we can make the separation
between the actor and the critic, because you know
we're gonna make actually two separate neural networks,
one for the actor and one for the critic,
but there will be the same encoding
of the images and the temporal relationships
for these two neural networks.
So this is the common part that we do
for these two neural networks.
This will be the same beginning for the two neural networks
but now things are gonna change for the actor and the critic
because we're gonna make one linear full connection
for the actor
and one differently linear full connection for the critic.
So let's take a quick break
and let's do that in the next tutorial.
Until then, enjoy AI.