-: Hello and welcome to this tutorial.
This special tutorial is gonna be super exciting
because we are getting closer to the A3C algorithm.
You're gonna see
that what we're about to implement,
and that is called Eligibility Trace, or Sarsa.
It is actually an algorithm of the asynchronous
active critics agents algorithms.
But we cannot consider it an A3C because
we're still going to have one agent.
But still, you're gonna see
that what we're about to implement
is actually taken from the following paper,
which is this paper: Asynchronous methods
for Deep Reinforcement Learning.
And it is in this paper that we'll find
the A3C algorithms that we we'll implement
as the final bonus of this course.
But as I said,
we are getting closer to it because the model
that we will implement right now is actually
this one: The asynchronous
and end step Q-learning.
That's the one.
So that's almost the A3C,
which is the one after that, but with one agent.
And the powerful thing about this
is this end step Q-learning.
We are going to learn the cumulative rewards
and learn the cumulative target
on n-steps instead of one step like previously.
And that's what will make the training
much more performant
and therefore our AI much more powerful.
So we actually have the pseudocode
for this algorithm.
It's this algorithm S2 right here.
So let's click on it, and there we go.
That's the algorithm we are about to implement.
But remember with only one agent.
The difference is that here they take an action
AT according to the epsilon-greedy policy based
on the Q values for the current state
and the action plate.
But in our case,
we didn't implement an epsilon-greedy policy.
We implemented in a Softmax,
but the rest is the same.
As you can see,
we are gonna compute accumulative rewards
on n-steps.
Actually 10 steps,
remember that n-steps is equal to 10.
And so we will implement this line of code
in our algorithm that we're about
to implement right now.
We're gonna get this
and mostly we are gonna implement
this as well.
You'll see that we will get
the maximum of the Q values
for the current states and the current action.
And this theta here, is just a target parameter.
So let's do this.
Let's attack this algorithm.
This one is called the asynchronous
and step Q-learning
but we don't have the right to say
asynchronous as far as we're concerned
because we only have one agent.
But therefore we can call it
n-step Q-learning Eligibility Trace,
or even Sorsa
All right, so let's do this.
It's gonna be pretty fun.
We can basically follow the pseudocode here
and that's what we're gonna do.
And so as you can see,
a parameter that we'll need is
a Gemma, the Gemma parameter.
That is the Dk parameter.
And therefore we will start
by introducing a variable
for this gamma parameter and choosing a value.
So let's do this.
We actually don't need a class to implement this.
We can simply implement this with a function
because you know,
we don't really need to create objects
for this Eligibility Trace model,
a function will be enough
because basically what we wanna do
is to return the inputs
and the targets
so that later when training the AI,
we are ready to minimize the distance
between the predictions and the targets.
And to get the predictions, we need the inputs
because we are gonna apply our brain
on the input to get the output signals.
That will be our predictions.
And then once we have our predictions
and our targets
we will be ready to train the AI
by trying to minimize the square distance
between the predictions and the targets.
So that's the whole point of doing this right now
we are implementing this function to be able to
return these inputs and these targets
so that we can be ready for the training to
minimize the square distance,
predictions minus targets.
All right, so let's do this.
As we said, we want to implement a function.
So we start with def, this function
we're gonna call it eligibility underscore trace,
you can also call it Sorsa,
you can also call it n-step clearing,
whatever you want,
but let's call it Eligibility Trace.
And this function is gonna take one argument
which is going to be a batch.
And why a batch?
It's because we're gonna get
some inputs and some targets
because we're gonna train the AI on batches.
And so the inputs and the targets will go
inside some batches, and therefore the input
argument here is this batch
that will contain several inputs
and then several targets that we will compute.
So, there we go.
That's the only argument we need.
Now, let's go inside the function
and let's define what we needed to do.
So as we saw in pseudocode of the paper
we need a gamma parameters.
So as we said,
we start by introducing this gamma parameter.
So gamma equals,
and we can already decide for value
and we are gonna choose 0.99.
That's a classic good value
for the gamma and Norris'.
I checked that this is a good value for our AI.
All right then, next step.
Next step is to prepare
our inputs and our targets
because that's exactly what we want to return.
We want to return the inputs
and the targets to prepare the training.
And so we can already initialize them
with an empty list,
because of course,
in these inputs inside the batch
we're gonna have several inputs all into a list.
And that's why I'm initializing the inputs
as a list as well as the targets.
There we go.
So we initialized our inputs and our targets
and in the end, this Eligibility Trace function
will return exactly these inputs.
And these targets will of course filter in.
We will have several inputs
and the associated several targets
in what will be returned by the function.
All right, next step.
Next step is to start a full loop.
And that's exactly
because we're following the pseudocode
of the paper.
This pseudocode, and as you can see,
there is this repeat code section.
And repeat is exactly a full loop
from pseudocode.
We are gonna compute
the cumulative reward right here
accumulated over the 10 steps.
And how is it computed?
Well, in each step, that is not the last step.
We're gonna get the maximum of the Q values
of the current state
we're in during this n-steps run.
And if we reach the last state of the 10 steps,
well, this will be equal to zero.
That is, we don't want to update it anymore.
And then we have this full loop
which is going to be another full loop.
They don't say repeat here, but that's the same.
It's going to be
a second full loop in our algorithm.
Well, we will update the reward this way
by multiplying it by the Dk parameter, Gemma
and adding the reward.
So let's do this.
Let's go back to Python
and let's start our full loop.
So for,
And what is going to be the iterator variable?
Well, that's going to be our 10 step series.
You know, our series of 10 transitions.
So we're gonna call this variable 'series'.
So that's represent a series of 10 transitions,
like sequence of 10 transitions.
So for series in,
And then what do you think?
Well, our series will belong to our batch
that is the batches on which we'll train the AI.
And so for series in batch,
that is for all the series
of 10 transitions in our input batch.
Well, what are we going to do?
Well to get accumulative reward,
you will see in psudocode
that we need the state
of the first transition of the series
and also the state of the last
transition of the series.
So what we have to do right now
is get these input states.
And so we are gonna put these two input states
into a variable that we're gonna call 'input'.
And we will get these two input states
the first one of the series
and the last one that's we're gonna put
into a NumPy array.
But no worries,
we will not stay with this NumPy array.
We will of course convert that
into a PyTorch variable.
But the first step is
to put these two input states,
the first one and the last one
into a NumPy array.
And so right here in this NumPy array,
we add the first input,
which is the input state
of the first transition of the series,
and that is series.
And then to take the first transition
we take the index zero of the series
that's the first transition,
and then we can access it
by taking its attribute, which is state.
And that's because in our experience replay file
we defined a special structure
for each of the transition.
And you know, this structure
each transition is composed
of a state, an action, a reward,
but then the last element, which is done.
So this special structure that we're allowed to
use right now comes
from the way we defined
a transition and experience replay.
All right, so with this
we get the input state of the first transition.
And now let's get also the input state
of the last transition of the series.
And to do this, that's the same.
We can just copy this
and paste it and replace the zero here
by the last index of the series,
which we can access
with this trick, minus one.
Series minus one that states
will get the input state
of the last transition of the series.
All right?
Then we need to put these two elements
inside some square bracket
because that's what is expected
by the NumPy array function.
And then an important thing to do,
since we are going to convert that into a torch
denser in a torch variable,
well remember a torch tenser is
by definition a special array
containing one single type.
And so we need to force having one single type
and as usual, we're gonna choose the float type.
And so I'm adding this parameter here
D type equals Np dot float 32,
can take this one,
and now we can convert that
into a torch tensor in a torch variable.
So let's do this.
To do this, well first let's convert
that into a torch tensor.
And remember, we can use torch dot from NumPy.
There we go.
And we put all the array of the two input states
inside this torch tensor
with the torch from NumPy function.
Perfect.
So that will convert this array
of the two input states into a torch tensor.
And now we put this torch tensor
into a torch variable using the variable class.
So input will be an object of the variable class.
And in fact, as you understood
this variable class takes all this
as an argument and that creates the object.
All right, so now we should be good.
We have our two inputs that we need.
That is the input state of the first transition
and the input state of the last transition.
And now, now that we have the input,
well, what can we get?
We can get the output signal
of the brain of the AI,
that is the prediction.
but we're gonna call it 'output'.
That's the output signal.
And to get the output.
Well, now that's very easy
because we already have a brain created
which is our convolution neural network.
And so we can simply take our brain CNN applied
to the input, which will return the prediction.
That is the output, as simple as that.
And now we are already ready
to move on to the next step.
And the next step is to start to
compute this cumulative reward.
So now we're gonna do exactly the same
as our S2 algorithm, the Sorsa,
or should we call it
N-steps Q-learning.
We are going to introduce
the cumule reward variable
which will be the cumulative reward.
And let's go back to the paper.
As you can see right now, what we have to do to
get this cumulative reward, which is R here, well
at each step of the 10 steps run,
we need to update it
by adding a zero to this cumulative reward.
If we reached the last state
of the series or the maximum of the Q values,
if we haven't reached the last state
of the series that is for all the steps
except the last step.
So let's simply implement this.
So let's go back to Python.
So this cumulative reward, as we just saw,
is going to be equal to
0.0 if we reached the last state.
And we can write this condition this way,
if series of index minus one,
that is the last transition of the series,
then we add dot done.
Because done actually is a attribute of, you know
this transition structure
that we defined in experience
replay, our experience replay file.
And this done comes from actually
the open AI structures
because if we go to the OpenAI Gym website
which is actually right here, I prepared it.
So that's the doom corridor of V zero.
And if we go to documentation
and then if we, so that's the tutorial
I really encourage you to have a look at it.
You can run an environment,
but mostly you can see
that our observations
that is are transitions are defined
by an observation, a reward, and this done here.
And this done means exactly
that a transition or a step is over.
And so we're gonna use this done here
for our if condition.
Therefore, if series minus one dot done means
if the last transition of the series
is over. Is completed.
And so this cumulative reward
is going to be equal to zero
if the last transition of the series is done.
And else, if,
we haven't reached the last transition,
well cumulative reward is going to be updated with,
as we said, the maximum of the Q values.
And since this output here
is the output of the brain,
that is the predictions of the neural network.
And as you know, the predictions
of the neural network are the predicted Q values.
Well, this output contains the Q values.
And since we need to take the max of the Q values
well we need to add first this index
because the structure contains the Q values
in the index one, and then we need to add data to
access the data of this output structure.
You know,
it has the special structure of a torch variable.
So with this, we get our Q values
and then we want to take
the maximum of our Q values
and so simply we add dot max
and now we get exactly what we want,
as in the paper.
This maximum of the Q values,
for the non terminal state st.
Perfect.
And so now what we're gonna do
is make this second for loop.
That is for the 10 steps of the series.
We are going to update
the cumulative reward this way:
By multiplying first by Gamma, the Dk parameter,
which we already have,
and then adding the reward.
So let's do this.
We are actually going to do exactly the same
as in the pseudocode.
-: As you can notice, they start from the right
so they're not starting
with the first step and going to the last steps.
They start with the last step T minus one
up to the first step, T start.
So that's exactly what we're gonna do
and that's because we want to get in the end
the cumulative reward being
equal to R equals R zero plus Gamma R one
plus Gamma squared, R two plus dot dot dot
plus Gama at the power of 10, R 10,
where R one, R two, dot dot dot R 10
R the reward obtained
in each of the end steps of the series.
So let's take a quick break
before attacking this second full loop
and I'll see you in the next tutorial.
Until then, enjoy AI.