-: Hello and welcome to this tutorial.
All right, so now we have our AI.
It is ready to be trained,
and the first step of the training
is to set up experience replay.
So we're slowly getting there, the training,
and the good news is that we have an implemented version
of experience replay besides that is adapted
to eligibility trace,
which I remind is a technique that
instead of learning the Q-values every transition,
learns it every 10 transitions.
So basically that's exactly the same as before,
but instead of having a single target,
a single reward for each step,
we're gonna have a cumulative target on 10 steps
and a cumulative reward on 10 steps,
and we will learn on the 10 steps each time.
So we are learning on 10 transitions,
10 steps instead of one like before.
And with this, our AI will work wonders,
and that will make some wonders for the training process.
The training will take much less time
thanks to this technique.
But we have to specify in experience replay
that we're learning every 10 steps.
So that's why this experience replay is not
a classic implementation of experience replay
like the one for the self-driving car.
It is an experience replay implementation
taking into account this 10 steps learning.
And therefore you will find
in this experience replay file two classes,
one class that makes your AI progress during 10 steps
so that it can sum the rewards observed on these 10 steps.
That's the first class.
And we need this class
because we need to include these 10 steps
in the replay memory class,
which is the class we implement for experience replay.
And that's how we make sure that the memory also takes
into account the fact that we're learning on 10 steps.
So that's why you will find two classes
in this implementation of experience replay.
But that's only to take into account
that we're learning on 10 steps
and that must be taken into account also in the memory.
So speaking of our memory, let's create it.
We're gonna call our memory memory.
And so memory is going to be an object
of the replay memory class,
and the replay memory class is a class
of this experience replay PY file.
And so I'm taking first this file experience replay,
then dot, and that's where I take the replay memory class.
Perfect. And now as you can see,
we have to input two arguments.
The first argument is n steps,
which corresponds exactly to the number of steps
on which we're gonna learn the Q-values.
So you know the number of steps on which we accumulate
the target and the reward.
So we're gonna have a cumulative target
and a cumulative reward.
And then the second argument is the capacity.
That is the size of the memory.
So, for example, here we can see 10,000.
So if the capacity is equal to 10,000,
that means that the memory will have a size of 10,000,
and therefore that means that we will get a memory
after 10,000 last steps performed by the AI.
But again, we're not gonna learn every transition.
We're gonna learn every 10 steps among these last
10,000 steps of the memory,
and that's exactly this new feature
that we introduced here compared to before.
Before we only had this replay memory trick,
and here we have this replay memory trick
plus this trick of learning every 10 steps.
And we're gonna learn every 10 steps,
and we're gonna do it in the memory
composed of the last 10,000 steps.
And this that is experience replay combined
to an eligibility trace with 10 steps will considerably
improve the training performance.
So let's input these two arguments.
The first one is n steps,
and that will be equal to, well, for now,
let's say n steps.
We will specify what n step is right after that.
It will actually be an object
of the other class of this experience replay file,
which is the n step progress class,
and that allows to make the AI progress during 10 steps.
And remember, during the 10 steps,
we will sum the rewards on the 10 steps
to get the cumulative rewards over 10 steps.
And that is exactly eligibility trace.
So now what we have to do is create this n steps here,
and we create it with the second class that we have
in this experience replay file,
which is n step progress.
So now we're gonna create n steps like this,
and this will be an object
of the n step progress class
that we take again from our experience replay file.
There we go.
So that's the n step progress class.
And now we have to input three arguments.
As you can see, we have to input the environment,
which is the Doom environment here that we imported.
Then the second argument is our AI,
and this will be, of course,
the AI that we built right here in the previous section.
And the last argument is n step,
and this, that's where we'll specify that we want 10 steps,
to learn every 10 steps,
that is, every 10 transitions.
So let's input these arguments.
The first one is the environment,
and that's Doom,
and all right,
then the second one is our AI,
and that we called it AI.
That's the one here.
So this is just the name of the argument
of the n step progress class,
and this AI here is our AI,
the one that we built.
And then the last argument is n step,
and that is equal to 10.
All right, so right now we are just taking into account
in the memory that there is a learning on 10 steps,
and this learning on 10 steps is called eligibility trace.
So we're really working on the advanced stuff here,
but remember that's because we're trying to beat Doom,
that's nothing like making a piece of cake,
so we need these advanced techniques to make it work.
So now we're almost ready before moving on to the next step,
which will be actually about implementing eligibility trace.
The only thing that we have to include
is the capacity, of course,
and that is, let's say, 10,000.
The memory will have a size of 10,000,
meaning that the memory will contain
the last 10,000 steps performed by the AI,
and that will allow us to generate some mini batches,
as you remember, with a sample function.
The memory contains 10,000 transitions,
but to train the AI,
we're going to sample some mini batches of 10 transitions,
not one compared to before,
10 transitions this time,
and we will sample these mini batches of 10 transitions
in the memory composed of the 10,000 last steps.
All right, so now I guess we're ready
to move on to the next step,
which is about implementing eligibility trace.
So we're gonna have some adventure here.
This will not be a simple implementation.
So have a good break,
and when you're ready, we can attack this.
Until then, enjoy AI.