Speaker: Hello and welcome to this tutorial.
So now the agent has done its expiration and then
what he's about to do is to update the shared network.
So, the first thing we're gonna do is initialize
the cumulative reward.
We're gonna call it R, capital R, and we will initialize it
as a torch tensor, but that will have dimensions one by one.
Because it's just a value, but we want it to be a tensor.
And so, I'm using here dot zeros and then one, one.
So basically the accumulative rewards is initialized
to zero, okay?
Then, same if we are not done, that is
if the game is not over, what we want right now
is the cumulative reward to be equal to the value
of the last state reached by the shared network.
So we're gonna get the value output, you know
the value of the V function, output of our model
and this is the value we will give to the cumulative reward.
So, let's first get this value.
We can get it this way value then, you know,
since we only want, the value we can add here
underscore and then underscore again.
And then we get our model because it'll output this value
but only the first output of the model.
Thanks to this double underscores here.
And here we can just copy paste what we have here.
That is, the inputs of the model with the inputs images
and the top all of the hidden states and the cell states.
So I'm just pasting that, and there we go.
We will get the value.
And so now, what we're gonna do is give to R this value.
So R will be equal to value
and to access to the value we add this dot data here.
All right.
Now, the if condition is done, and now what we're gonna do,
since we just got a new value by, you know
getting the output of the model,
the first output of the model.
Well let's already append this new value
to the values lists.
Therefore, we can take directly our values list then
dot append, and we input variable R.
Because R contains this last value.
So great, that is done.
Now we are going to initialize the losses
and remembering the intuition lectures.
You have two losses.
You have the loss of the policy that is the loss related
to the predictions of the agent.
And then you have the loss of the value, which is
the loss related to the predictions of the critic.
So we are going to introduce these two variables
and initialize them to zero.
And therefore I'm gonna take here policy, first
variable policy loss, initialize it to zero
and then value loss, the loss of the value
and same initialize it to zero.
Then let's not forget to set the cumulative reward
as a torch variable.
Because we will need it to be a torch variable, because
we will be computing a gradient with respect to it,
because the cumulative reward is going to be a term
of the value loss.
So with this variable, it is now attached
to the dynamic graphs with a gradient.
And now finally, the last thing we need to do
before starting the big training loop, you know
when we apply stochastic gradient descent,
to reduce this loss between the predictions
and the targets.
Well we need to initialize the GAE.
The generalized advantage estimation
and not gated auto encoder, be careful with that.
GAE, the variable that we're about to initialize right now
is generalized advantage estimation.
So as a reminder, generalized advantage estimation
is by definition the advantage
of playing the action A by observing the state S.
So it's a function of the action A and the state S
and it is equal to the difference
between the Q values, QAS, and the value of the V function.
So actually I can write it here.
The generalized advantage estimation is a function A
of the action and the state S, and that is equal
to the Q values of the action A and the state S,
minus the value of the V function applied to the state S.
That's the generalized advantage estimation
and that's what we want to initialize right now,
and we will initialize it to zero.
But it has to be torch tenor,
so we're gonna use the same trick
as what we just did right here.
We are gonna take the torch library
and apply the zero's function, to set it
as a tenor of only one value, which is zero.
And we are going to introduce this new variable GAE
and that will be equal to that torch
dot zeros one one, as initialized to zero.
So this will be initialized to zero
and therefore the Q values of the action A
in the state S, will be equal
to the value of the V function of the state S.
All right, and now we are ready to start the fall loop.
So we're gonna have some adventure here,
so take a good break
and I'll see you in the next tutorial, to attack that.
Until then, enjoy AI!