-: Hello and welcome to this Python tutorial.
All right, so now we're gonna make the forward function,
which will propagate the output signals of our brain
to the body of the AI
so that it will play the right action
to reach the vest.
But there is no right action yet
because there is no training yet.
We have not trained the AI yet,
but this is exactly what we will do in part two,
Implementing Deep Convolutional Q-Learning,
which, by the way.
I will rename Training the AI
with Deep Convolutional Q-Learning.
But right now we need to forward the signal
from the output layer of the brain to the body,
and that's exactly what we're gonna do
with this forward function,
which is the last function of our body.
So let's do this.
We start with def forward,
and according to you,
what argument is it going to take?
Well, it's going to take, of course, first self,
and then is there another one?
Well, yes, there is.
And what is it going to be?
Well, very naturally,
we want to forward the output signal of the brain
to the body
and therefore the input will be
the output signal of the brain.
And so now we need to give a name to these output signals.
And so I'm gonna add here the argument output.
All right, so that corresponds to the output signals
of the brain after the input images
are propagated through all the brain
to reach the output layer,
which is X here,
returned by the forward function of the brain.
And now this output signal of the brain
will be forwarded to the body
with this new forward function
that we make in the softmax body class.
So let's do this.
Let's add some column here.
And now, as you understood,
we're gonna use a softmax method to play the action.
That means that the body of our AI
after receiving the output signals of the brain
will play the actions with a softmax technique.
So basically now what we have to do is exactly the same
as what we did for the self-driving car.
We're gonna get our distribution of probabilities.
That's the first step.
And then we're gonna sample an action
according to this distribution of probabilities.
So basically what we could do now is get
our self-driving car file
and copy-paste what we implemented
for the select action function in the self-driving car.
But let's do it again.
It will be good practice.
And actually, you can try to type it before me.
Okay, so first what we're gonna do is get our probabilities.
So I remind this is a distribution of probabilities
for each of the Q-values
which depend on the input image and each action.
So we have one Q-value
for each of the six or seven possible actions,
and therefore we get a distribution of seven probabilities.
I'm saying seven
because I think there is seven actions instead of six,
because besides moving forward, left, right, or shooting,
we can also run.
So that makes seven possible actions,
and therefore we get a distribution of seven probabilities,
one for each Q-value associated to each action.
So props equals,
and now remember what we have to do?
Well, basically we have to use the softmax function
from the functional module.
So that's very simple.
We take our functional module first, then dot,
and then we take our softmax function.
Here it is.
We press Enter,
and now we input the arguments of the softmax function,
which I remind are the elements for which
you wanna create a distribution of probabilities.
And so that's, of course, the Q-values,
that is, the outputs of the neural network.
That's the outputs of the neural network
for which you wanna create a distribution of probabilities.
And I remind that we want to create this distribution
of probabilities to be able to explore the different actions
instead of directly picking the one
that has the maximum Q-value.
If we directly pick the one that has the maximum Q-value
while we don't explore much the other actions,
and we might miss something.
But with the softmax method
we can do some more exploration
and therefore maybe find some hidden solutions
in the patterns that might be much better.
So again, I highly recommend softmax,
and therefore now what we have to do is input the Q-values,
that is, our outputs here,
the outputs of our brain.
So outputs, there we go.
But then we have this temperature parameter that we can use
that we can configure to customize the exploration.
Remember that the higher we set the temperature,
the less exploration of the other actions we will do
because the best action will be selected
with the higher probability
as opposed to the other actions
which will be selected with lower probabilities.
So that's exactly like for the self-driving car,
and therefore we have to multiply the output here
by our temperature parameter, cel.t.
There we go.
Perfect. Now we get a little warning
because we haven't used perhaps yet,
but we are about to use it now.
And so that brings us to the next thing we have to do.
How are we gonna use these probabilities?
Well, we're going to sample the final action to play
from this distribution of probabilities,
and therefore, what we have to do now is use
the multinomial function to sample the action
according to this distribution of probabilities.
So now we're ready to get our actions.
So I'm creating a new variable here because that will become
the actions that will be played by the body of our AI.
And so now we take our distribution of probabilities, probs,
to which we add dot
and then the multinomial method.
All right, and now we get our final actions to play.
They're assembled from our probs distribution.
Okay, perfect.
So now we are ready to return what we want,
that is, the actions to play.
And these are, of course, actions.
And now the warning should disappear.
We use everything we want.
There we go. Perfect.
So now the forward function is ready,
and congratulations, the body is also ready.
So now we have our brain, we have our body,
and therefore we're ready to assemble them
to make the future AI.
Our future AI will be composed of nothing else
than a brain and a body,
and so it will have intelligence
and a body to play the actions,
which will be the right actions to play
thanks to its intelligence.
But remember, before we have to train its intelligence,
and that's what we'll do in part two,
Training the AI with Deep Convolutional Q-Learning.
All right, so let's make the AI in the next tutorials.
It's again going to be a class of two functions, I think.
And so this will require two or three tutorials.
So I can't wait.
This will be exciting.
And until then, enjoy AI.