-: Hello and welcome to this Python tutorial.
All right, so in this tutorial, we're gonna make
the function that will select the right action at each time.
So basically we're gonna implement the part
that will make the car do the right move at each time,
that is going left, going straight,
or going right to reach the goal.
And to avoid the obstacles that is decent.
So let's do this right now.
We are gonna start as usual with a def to define a function.
And then we give a name to our function,
which we're gonna call select action.
Then some parenthesis.
And this select action function will take two arguments.
The first one is self, as usual, to refer to the object.
And a second argument, which,
according to you is going to be which one?
Well, what could it be?
If you think about it, the action we select comes
from the outputs of the neural network.
Because the outputs of the neural network are the q-values
for each of the three possible actions.
And therefore, the action that will play,
the action that will be the output of the neural network,
depends on the input state.
And the input state is exactly the second argument we need
for the select action function.
It's because we are literally going to take the output
of the neural network,
and of course the output of the neural network
directly depends on the input of the neural network.
So that's gonna be our argument.
And now we can give it any name.
We will actually call it state.
Because the inputs of the neural networks
are the input states that are encoded
by vector five dimensions.
The three signals, orientation, and minus orientation.
And so now, things are gonna be easy.
We are gonna feed the input states into the neural network,
the one that we built right above.
Right here, with the network class.
And then, then we're gonna get the outputs.
Which are the q-values
for each of the three possible actions.
And then using the softmax method,
which I'm going to explain in this tutorial,
we are gonna get the final action to play.
So let's do this.
Let's go into the function and let's implement all this.
So, the first thing we need to start with is
about what I've just mentioned.
Softmax.
The idea of the softmax is that we're gonna try to
get the best action to play at each time.
But, at the same time,
we will be exploring the different actions.
And how can we do that?
How can we get the best action to play
while still exploring the other actions?
Well, we use this idea of softmax.
Which consists of generating a distribution
of probabilities for each of the q-values, q states action.
You know we have one q-value for each action.
Go left, go straight, or go right.
But this q-value also depends on the input state.
That's exactly the Q-function you saw
in the intuition lectures.
This Q-function is a function of the state, and the action.
So since we have here one input state,
which is the state here.
And three possible actions.
We have three q-values.
Q, state action one,
q state action two,
and Q state action three.
And we are gonna generate a distribution
of probabilities with respect to these three q-values.
That is, we're gonna have one probability
for the first q-value.
One other probability for the second q-value.
And a third probability for the third q-value.
And all these three probabilities will sum up to one.
And so, we're gonna do all this with softmax.
And softmax will attribute a large probability
to the highest q-value.
That's why an alternative to softmax is a simple argmax.
You know, directly taking the maximum of the q-values.
But in that case, we're not exploring the other actions.
Thanks to these probabilities
we can explore somewhere else using a temperature parameter
that we're gonna see very quickly.
We can still explore them
by configuring this temperature parameter.
That's why in general, for deep Q-learning,
I highly recommend to use a softmax
rather than a simple argmax.
All right, so let's implement softmax,
and therefore as you understood,
since softmax returns the probabilities of each
of the three q-values for the three possible actions,
where the first variable we're going to create is probs,
referring of course to these probabilities.
So probs equals.
And now we're gonna take our softmax function.
And according to you, where are we going to take it from?
Well, of course, remember we imported
the torch.nn.functional submodule.
Which I remind is the module that contains most
of the actions to implement in neural network.
We gave it the shortcut F.
And so that's exactly from this functional submodule
that we're gonna take our softmax function.
But, since we gave it the shortcut F,
we start here with an F representing functional.
From which we take our softmax function.
Here it is, that's the first one.
And parenthesis.
All right.
And now, what do we need to input in this softmax function?
Well, that's of course the entities
for which we want to generate the probability distribution.
And where are these entities?
Well, these are of course the q-values.
So now the question is, how can we get the q-values?
Well, of course the q-values are the output
of the neural network.
And to get these outputs of the neural network,
well, here we go.
We need to take our neural network.
But in fact, we already have it.
Because that's what we initialized in the innate function.
You know, we created self.model,
which is nothing else on our neural network.
Because it is an object of the network class.
And so that's perfect.
We can just take our model here in softmax.
Apply this model to the input state.
Which is the argument here.
And that will return the outputs that we're looking for.
That is the q-values.
And so now your intuition, why we had to take the model here
to introduce it in the innate function might get better.
For those of you starting with object oriented programming,
you will see that all this will become natural.
So softmax, then, so we take our model, self.model.
Because this must be the model
of the object that we created here.
But then, we need to get the output
of our neural network model.
And therefore we're gonna add here some parenthesis.
In which we're going to input, well,
the input state, named state here.
So, what we wanna do at first is enter state.
But now we must be careful to something.
State looks like a simple state right now.
But remember, that state is actually going to
be a torch tensor.
Because later we're gonna use
this self.last state,
to put it as the argument of the select action function.
The state argument that is here is actually
going to become later this self.last state.
And since this is a torch tensor,
well the model will accept it.
So that's fine.
But now we can improve the algorithm.
So. as you understood, state is a torch tensor.
And, as we said earlier, most of the tenors are wrapped
into a variable that will also contain a gradient.
So right now what we're gonna do, first,
is wrap this input state that is a tensor,
into a torch variable.
But since this is the input state,
well there is not going to be some differentiation.
We will not be using the gradient
of this state torch variable in the computations.
And therefore, what we're gonna do now is convert
this torch tensor state into a torch variable.
Like so.
But then to specify that we don't want the gradient
in the graph of all the computations of the nn module,
well, we will adhere, comma, volatile, equals, true.
So that now we have our state torch tensor
into a torch variable.
But, thanks to this volatile equals true parameter,
well, we won't be including the gradient associated
to this input state,
to the graph of all the computations of the nn.module.
So that's another technical trick.
This will save us some memory,
and therefore this will improve the performance.
So I highly recommend to do this.
And now we're gonna add something more fun.
It's about this temperature parameter
that I've just mentioned.
So this temperature parameter is the parameter
that will allow us to modulate how the neural network
will be sure of which action it should decide to play.
So, this temperature parameter will be a positive number.
And the closer it is to zero,
the less sure the neural network
will be when playing in action.
And the higher this temperature parameter is,
the more sure the neural network will be
of the the action it decides to play.
And to add this parameter,
I'm going to multiply the output,
which are the q-values,
by this temperature parameter.
So let's start, for example, with seven.
And I'm going to specify here
the little comment, t equals seven.
So that's the temperature parameter
that I'm setting equal to seven.
We're gonna try some other ones,
but I just wanna start with a small one.
Because you're gonna see that with a small one,
our car will still behave like some kind of an insect.
But then by increasing this temperature parameter,
our car will look more like a car.
And besides, the self driving will be much, much better.
And so that makes sense,
because the higher is this temperature parameter,
the higher will be the probability of the winning q-value.
Because for example,
if we have softmax of the q-values,
let's take some simple numbers, 1, 2, 3.
If softmax of 1, 2, 3 equals, for example,
0.04, 0.11, and 0.85.
Then by increasing the temperature, you know,
by taking a higher temperature.
Right now the temperature equals one.
By taking a higher temperature, like for example, two.
So, softmax, let's copy this and multiply it by,
for example, two or three.
Softmax have the same q-values,
but multiplied by this temperature parameter of three.
Well, we will get something like zero for the first q-value.
Because this had a very low probability.
So that's something around zero.
Then, something very small for the second probability,
because this was still a low probability.
So let's say, for example, 0.02.
But then, the third probability,
since it was the largest one,
and a pretty high one/
Well, by increasing the temperature,
this probability will be even larger.
Because we're gonna be even more sure
that this is the right q-value corresponding
to the action we must play.
And therefore, this is gonna be something like 0.98.
Now, by increasing this temperature parameter,
well we are now even more sure
that the third action here should be the action to play.
Because the probability for the q-value
of this action is not only the largest one,
but also very high.
So that's what this temperature parameter is all about.
It's about the certainty
of which action we should decide to play.
All right, so, I'm gonna remove this comment.
This was just to explain.
And now, let's get our action.
So, how are we going to do that?
Well, the principle of this softmax method is
not only to generate a probability distribution
for each of the q-values, but also,
and that's the second step of this softmax method.
We take a random draw from this distribution
to get our final action.
And of course, we will will have a high chance to
get the action that corresponds
to the q-value that has the highest probability.
Because that's exactly how a distribution works.
So there we go.
Let's get our action.
So we're going to introduce a new variable
that we're gonna call action.
And this action is going to be a random draw
of the probability distribution
that we just created at this line before.
And so how do we get such a random draw?
Well, we're gonna take our probs,
probabilities of each of the q-values.
We take probs, and then dot,
and then we're gonna use the multinomial function.
And that will give us a random draw
from this distribution, probs.
So that's all.
That will give us the action.
Perfect.
And now of course, we are going to return the action.
But there is a little trick here.
Well, it's the fact that this probs dot
multinomial returns the pie torch variable
with a fake batch.
You know, this fake dimension corresponding to the batch.
And therefore to get the right result that we want,
that is the action zero, one or two.
We just need to add here, data, and then some brackets.
And the action zero, one, or two that we're looking for
is contained in the indexes, zero and zero.
All right, and there we go.
Now we have our action.
Thanks to this select action function,
the AI will now know which actions are play at each time.
Terrific. So now we can move on to the next function
which will be the learn function.
And that's where we will train the whole neural network
you know, with all the forward propagation and
then the back propagation,
using stochastic gradient descent.
Well, basically, we will implement the whole training
of the deep learning model that is
at the heart of our artificial intelligence.
So I can't wait to do that.
This is going to be an exciting tutorial.
And so, I'll see you in the next tutorial.
Until then, enjoy AI.