Speaker: Hello, and welcome to this Python tutorial.
Now we're gonna make the forward function,
that will forward propagate the signal,
throughout the brain, from the very beginning,
with the input images up to the outputs,
which will contain the Q values for the actor
and the V value.
That is the value taken by the V function, for the critic.
So, it's gonna be quite similar as what we did for doom
but this time something is gonna change.
The main thing that's gonna change is that,
now we have an endless TM, in the brain.
So, we'll have to do something more,
to propagate the signal and be careful with that.
And the other thing, less important,
but still that changes compared to before,
is that we're not gonna use a ReLU activation function,
as you know, the non-linear activation function.
But we're gonna use the ELU,
which is kind of a more, sophisticated ReLU function.
We will see that in the pytorch documentation.
So let's do this.
Let's make this function.
We start with the depth.
It's actually the last function of this (indistinct) class.
So, we're gonna call it forward like last time.
And this forward function is gonna take self, the object,
because we're gonna use the object, and the inputs.
So, important to understand what will these inputs be.
This will not only be the input images,
these inputs will also contain the hidden nodes
and the cell nodes of the LSTM.
So, that's why I wanted to highlight,
that some things are gonna change now.
Basically, we're considering, in the forward function,
the hidden nodes and the cell nodes of the LSTM.
And speaking of them, now, what we're gonna do is,
separate these two inputs of this argument inputs,
in the forward function, and how can we separate them?
Well, we can recall a new variable,
which will be the input images.
So, that's, this time, the input images.
And, we separate them with the topple HX and CX,
which is the topple of the hidden states
and the cell states, of the LSTM.
So, HX are the hidden states, and CX are the cell states.
All right?
And that will be equal to the input,
that is this argument here.
So now we've made this separation, and therefore,
we can start to propagate the signal throughout the brain.
And to do that, we are going to get, successively,
the different layers, from the first one to the last one,
by using our connections.
That is the convolutions, the LSTM connection
and the linear connection here, the full connections.
So let's do this. Now, it's gonna be the same as before.
We're gonna get our first layer, that we're gonna call X.
And to get this first layer,
we need to propagate the signal,
from the inputs, to this first layer.
And therefore, we need to use the first convolution.
Because it is this first convolution that propagates
the signal, from the inputs images, to the first layer.
So what we're gonna do now, is copy this,
because this is the first convolution.
We base that here, and we apply this first convolution
to our input images, which are now the right inputs.
And there we go.
That propagates the signal,
from the input images to the first layer.
But now, remember, we have to use
a non-linear activation function
to break the linearity,
in order to be able to learn
the non-linear relationships, inside the images.
And to do this, we are gonna use, as we said,
the ELU activation function, that we're about to
see right now, in the pytorch duct.
But before that, let's get it.
So to get it, it's like ReLU,
we take the functional module,
which has a shortcut, F, then dot, and then ELU.
And then we put all this in parenthesis
because we want to, non-linearly, activate the neurons
of this first layer here.
That we obtained, by applying the first convolution
on the inputs.
So now let's go to the pytorch duct,
to understand what ELU is.
Here it is.
So, you can access it to pytorch.org/./nn.html,
and then you have to find non-linear activations.
And in the non-linear activation function,
you will find, well ReLU, that's the classic one we know.
So it's just a max of zero and x.
You have to graph in mind.
Then you have ReLU six, which is this one.
So, a little more sophisticated.
And there we go, we have ELU.
And as you can see,
ELU is a ReLU, plus an additional element.
So, it's like a more sophisticated ReLU.
And so that's the one we use
to non-linearly, activate the neurons
in the different layers.
And by the way, this ELU activation function,
is called, the exponential linear unit.
So there we go.
We apply the ELU on the first convolutional layer
and now, things are gonna be easy.
We're gonna proceed to the next,
forward propagation of the signal,
which is from, the first convolutional layer
to the second convolutional layer,
which we are gonna call X
because, basically, we're just updating X.
Now X is the first convolutional layer.
And by propagating the signal,
from the first convolutional layer to the next one,
X will become the next convolutional layer.
And so to propagate the signal,
from the first convolutional layer to the second one,
we can simply, copy this and paste that here
and replace conf one by conf two.
And now, of course, the second convolution
is not applied to the input images,
but to X, that is the first convolutional layer,
which is right here.
All right, perfect.
Now we get our second convolutional layer.
And now let's propagate the signal, again,
from the second convolutional layer, to the third one.
And, therefore, we can directly copy this
and paste that here.
And replace comf two by comf three.
There we go.
And last one, now to propagate the signal,
from the third convolutional layer, to the fourth one.
And last one, we can just copy this again, paste that here.
And replace comf three by comf four.
There we go.
So, let's recap.
We start with our inputs.
We apply the first convolution
to get the first convolutional layer.
Then we apply the second convolution
to the first convolutional layer,
to obtain the second convolutional layer.
Then we apply the third convolution,
to the second convolutional layer,
to obtain the third convolutional layer.
And finally, we apply the fourth convolution
to the third convolutional layer,
to obtain the fourth convolutional layer.
And that's how the signal is propagated,
throughout the eyes of the ai.
So there we go.
We have, now the output signal, after the four convolutions.
And now you know what to do.
We need to expand this whole output signal
in one dimensional vector.
That's the flattening step.
So, we're gonna update X again.
X now will become this flattened one-dimensional vector.
And to do this, that's the same, we need to take X,
which is, so far, the fourth convolution layer, X, dot.
Then we use the view function,
and we first input minus one,
to say that, we want a one dimensional vector.
And then as a second argument,
we need to input the number of elements in this vector.
And that is, remember, 32x3x3,
and therefore, we input here 32x3x3.
There we go.
Now we have our flatten vector
and the flattening step is done.
Perfect. Now let's take care of the LSTM part.
So, as you understood,
the LSTM takes as input, the flatten vector,
this one dimensional vector of 32x3x3 elements.
So, it's already ready and well prepared for the LSTM.
The LSTM is now ready to take,
this flattened vector as input.
And, therefore, we can take our LSTM
and input, as argument first X, already flattened vector.
That is, this X right here, that we just expanded.
But, also, and that's where this topple comes into play.
We need to input HX and CX, and we can use HX and CX here,
because we made that separation,
from the original input argument of the forward function.
So LSTM X, the flattened output vector,
after the four convolutions
and this topple of the hidden and the cell node.
So there we go.
Then we must not forget the self,
because LSTM is a variable of our end function.
So a variable attached to the object.
So cell.LSTM.
And this, will actually, return two outputs,
a topple of two outputs,
which will be the output hidden node
and the output cell node.
So it's actually a topple.
And, therefore, we can update HX, the hidden node
and CX, the cell node, because that's exactly
the output of this LSTM here.
Great. So, we are almost done.
Now that we have the output of the LSTM,
we need to get the useful output
because, actually, only the hidden nodes are useful
and, therefore, we're gonna get it,
by updating X again and X will now be equal to HX.
It's the first element of the output topple
of the LSTM, X equals HX.
And we're almost done.
Remember that we have two brains, one brain for the actor
and one brain for the critic.
And, therefore, we have two output signals to return,
the output signal of the actor
and the output signal of the critic.
And, therefore, now what we're gonna do,
is return these two output signals.
And how can we do that?
Well, that's very easy.
We simply need to take our linear full connections
but separately, that is a linear full connection
of the critic and the linear full connection of the actor.
And we apply each of these full connections to the output X.
That is the useful output of the LSTM.
And that will be all.
That will be the output signal.
So there we go.
Let's do it.
We first take self, our object,
then we get the linear full connection of the critic,
which is critic and core linear.
Which we apply to X, the output signal of the LSTM.
And then same, we take self again, then dot.
And then we take the linear full connection of the actor,
which is actor and core linear, which same we apply to X.
There we go.
So that's the main thing we need.
But then we're also going to return
the topple of HX to hidden nodes and CX to cell nodes,
because we'll be using them later
in the retro loop of the LSTM.
All right, perfect.
So, now we're done with the brain,
or should I say the brains?
Because we actually made two brains.
One for the actor and one for the critic.
So, congratulations for making the A3C brains.
I hope that wasn't too overwhelming,
to combine a CNN and LSTM.
But at least, the good news is that,
we're really working with the best and most powerful model.
So, there we go.
We are actually done with this first foul model, the py.
And so in the next tutorial,
we'll take care of the optimizer
because we're gonna make a separate optimizer.
We're not going to code each line of code
because a lot of it comes from the research papers.
And this is actually pretty specific.
And if we go into the deep details of what's going on
with this optimizer, this might be a little too overwhelming
for what's gonna happen next.
Because we still have the train function to make,
which will be a huge function,
and that contains the algorithm of the A3C.
So trust me, you want to keep some energy for that.
And, therefore, we will not spend too much time on this.
But still, I will expand the code
and you will understand the whole idea
behind this optimizer.
So, congrats again, for making this act to critic class
and I'll see you in the next tutorial to make the optimizer.
Until then, enjoy ai.