-: Hello and welcome to this tutorial.
In today's tutorial,
we are going to synchronize with the shared model.
So what we're gonna do
is still saying the function, of course,
and then initialize the length of one episode.
So we're gonna call the length of an episode,
episode on this core length.
There we go.
And we are going to initialize it to zero
but then this episode length will be incremented.
And speaking of incrementing it,
that's exactly what we'll do.
So we're gonna use a while loop and use this trick to say,
while true colon, to repeat what's gonna happen now,
what's gonna happen inside this while loop.
And so the first thing that's gonna happen
in this while loop
is this incrementation of the length of an episode.
So the first thing that we're gonna do
is increment it by one.
And to do so, we can simply take episode length
and add here, plus equals one.
And now, we are going to synchronize with the shared model.
That means that it is now that the agent
will use the shared model to do its little exploration
on a certain number of steps.
And how is the model going to get this shared model?
Well, we need to take our model then dot,
and then use the load state dict method
because we're gonna use it to get the state dictionary
of our shared model.
So we have to put the shared model first
and apply then the state dict method to get the parameters
of the shared model.
And that's how our model here will get the shared model
to do its little exploration.
Okay, and once the model gets this shared model,
now we have to distinguish two cases.
The first one is if done, meaning if the game is done.
So if the game is done, then what happens in that case?
Well, we have to reinitialize the hidden state
and the cell state of the LSTM in the model.
And so that's why now I'm gonna take cx, the cell state
and also hx, the hidden state,
and I'm going to reinitialize them both.
And how we're going to reinitialize them?
Well, with only zeros.
There will be a vector of 256 zeros
because remember, the outputs of the LSTM
has dimensions 1 and 256.
So there we go.
We're going to initialize them by using the torch library
then the zeros function.
And since we want a vector of 256 zeros,
we are gonna input here the dimensions one
for the vector, and 256 for the number of elements
which will be zeros.
And there we go.
But then we will convert that into a torch variable
because then some gradients will be computed.
So we need to integrate this with a gradient.
All right?
And we're gonna do the same for the hidden state just below
and reinitialize them the same way.
There we go.
So that's if the game is done.
And now, the other case which we can access with else.
Else, then what happens in that case?
Well, we're gonna keep the old cell states
and hidden states.
And so very easily, we can keep the old ones this way
by typing cx equals variable, cx dot data.
And same for the hidden states,
we can simply add here hx equals variable, hx dot data.
All right, good thing done.
Now we can get out of the else
because we are basically done with these two cases,
whether the game is over or not.
But we stay in the while loop
because now, we're gonna do some more things
which basically are all the training process.
And so what we're gonna do now
is initialize several variables,
which are gonna be at the heart of the computations
in the training.
So let's do this.
We're gonna need the values,
which remember is the output of the critic.
So that's the v function
and we will initialize them as an empty list this way.
Then we're gonna need the log probabilities, so log probs,
and we will also initialize it as an empty list.
Then, of course, we're gonna need a reward
that we will also initialize as an empty list.
And finally, we're gonna need the entropies.
Something new but this is indeed at the heart
of the training computations.
So empty list as well.
So now that we initialize these four variables,
we can start a new for loop.
And then this new for loop,
we will update the values of these four variables.
And so this new for loop is gonna be a for loop
over the exploration steps.
And therefore, the looping variable
is going to be our steps.
So for step in range
and inside, we can directly input params dot numsteps
because params dot numsteps is exactly the number of steps
of the exploration.
So for all the steps in the exploration, what do we do?
Well, we are gonna get the predictions of the model.
You know, what is returned by the model.
And to get these predictions, we can simply take the model
and apply it to the input.
So that's the input signal.
It goes through the brains in the model
and that will get us the outputs,
but it will get us several outputs, you know?
It will get us the values of the v function
which is the output of the critic.
Then the Q-values, QSA, which is the output of the actor.
But also don't forget that it will also output
the tuple of the hidden state and cell state.
Because remember, if we go back to our model,
well, in the forward function,
we can see that indeed it returns the output of the critic,
that is the value of the V function, VS,
then the output of the actor, which are the Q-values, QSA,
and also the outputs of the LSTM,
which is this tuple hx and cx,
the hidden state and the cell state.
So we must be careful with that.
This is quite different than what happened before.
And therefore, we're now going to apply the model
to the input, which is a state.
But now there are several things to do,
which are related to torch, but that gives, of course,
power to what we're doing.
The first thing we need to do is to unsqueeze the state
to add this fake dimension that must have the index zero.
That's because the model can only accept
a batch of inputs and not an input by itself
in a vector or a tensor.
So that's the first thing we must do, the unsqueeze,
but then that's not all,
we need to convert our input state into a torch variable.
So I'm adding here the variable.
So now we're okay with the state, the input state,
but remember that the inputs of the forward functions
are actually the inputs image,
so that's what we just took care of,
but also this tuple of hx, the hidden states,
and cx, the cell states.
And therefore, we need to add here
this second part of the input with the tuple of hx and cx.
All right, and we must take care of the parenthesis.
There we go, we have our two inputs.
The first one is the input state.
That is the input images,
all converted into a torch variable
and unsqueezed to add this fake dimension of the batch,
and this tuple of the hidden state and the cell state.
So we're all good to go.
We are ready to get our predictions.
And now, since this return, well, our three predictions,
the output of the critic, the output of the actor,
and the tuple of the hidden state and the cell state
by the LSTM,
well, we're going to introduce some three new variables now
which will be these three outputs.
So there we go.
The first output is the value of the v function
which is the output of the critic.
So we're gonna call it value.
So there we go.
That's the first output.
Then the second output is going to be
the output of the actor, and that's the Q-values, QSA.
But since the Q-values are associated to the actions,
we can also call them the action values.
All right.
And then final output returned by the model
that's the tuple of the hidden state, hx,
and the cell state, cx.
And there we go.
We have our three outputs returned by the model.
Perfect.
So now that we have the predictions,
we need to use a softmax to play the right action.
And so now that's gonna be exactly the same
as what we did before.
The next step is to get our probabilities
so we can call them prob.
And that's where we use the softmax method,
which we take from the functional module
that has the shortcut f, so f.softmax.
And that will generate a distribution of probabilities
of the input that we're about to input right now,
and which of course, the action values,
that is the Q-values, that is the output of the actor
in the model.
Okay, so now we have our probabilities, but as you noticed,
we're gonna work with the entropy.
And to get the entropy,
we not only need the probabilities,
but also the log probabilities
because the entropy is the sum of the product,
log prob times prob, all this multiplied by minus one.
And so we also need to get our log prob,
which same are going to be generated from log softmax.
So instead of taking a distribution of the probabilities,
we take a distribution of the log probabilities,
and that we do it with the log softmax function
to same, we apply to the Q-values
which we call the action values.
All right, so now we have the prob and the log prob,
and so we are ready to get the entropy.
And the entropy,
what is the formula for that?
Well, as I just mentioned, we take the log prob,
we multiply it by the prob,
then we're gonna take the sum of all this.
And to do that, we can add here dot sum one.
We actually used the streak many times now.
And as we said, we multiply all this by minus one.
So it's the minus of the sum of the product,
log prob times prob.
Perfect.
And now, we are gonna store this entropy
that was just computed in our list of entropies,
because there we go,
we have the last computation of the entropy,
and so we need to store it in the entropies list.
And to do this, nothing more simple,
we're gonna use the append function, of course,
because entropies is a list.
So we take our entropies list then dot
and we use the append function to add the entropy
that was just computed.
All right, so we're gonna take a break now.
We're gonna do this step by step.
In the next tutorial, we will play the action
by taking a random draw of this generated distribution
of probabilities.
And after we play the action,
we will get the value of the state,
and we will eventually store our new transition,
state, reward, and done.
So that will be a new big step done
and we will complete that in the next tutorial.
Until then, enjoy AI.