-: Hello and welcome to this Python tutorial.
So now that we are ready to train the network
to minimize the square distance
between the outputs and the target,
thanks to what we did with eligibility trace
in the previous section.
Well, basically, we are ready to start the whole training
by, you know, getting our inputs, our targets,
our predictions, then computing the last error
between the predictions and the target,
and then doing the backward propagation
was to get the gradient descent to update the weights.
So we are ready to do all this,
but since we want to compute the moving average
on 100 steps, you know, to keep track of the average
during the training.
Well, just before we do this whole training,
we are going to make a class right now
that will get this moving average on 100 steps.
So no worries, we will do it quickly.
We will make a class with three functions,
but we will do all this in this single tutorial.
So we will do it quickly.
We already did it, and besides,
we want to focus on the training right now
because that's the most important.
So let's make this class right now and this single tutorial.
All right, so we are going to introduce a new class
which we're gonna call MA for moving average.
And then here we go with our first function.
So that, of course, the init function,
that never changes, init.
And this init function is going to take two arguments.
The first one is self for the moving average,
future object, and size,
which will correspond to the size of the list of the rewards
of which we're going to compute the average.
So this is going to be 100.
All right, so we have our arguments for the init function,
and now, let's go inside the function.
Now, you know what to do.
We have to initialize the variables specific to the object.
And these are, well, first,
the first one is going to be a list of rewards.
So that's gonna be the list containing the 100 rewards
of which we're going to compute the average.
So here right now, we're just simply initializing this list
with this empty list here.
So list of awards,
and then the second variable of our future object
is going to be, of course, the size.
And the size is going to be equal to the argument
we will input when creating
the future moving average objects.
So size here,
and already, we are ready to move on to the next function,
which is going to be the add function.
And that will add the cumulative rewards.
Be careful, it's not the simple reward,
it's the cumulative reward,
and that's because, you know, we are doing eligibility trace
and therefore, learning every 10 steps,
and therefore, learning with cumulative rewards
and not a simple reward.
So this add function that we're about to make
will add the cumulative reward to that list of rewards.
So depth, we're gonna call it add, of course.
And this add function is gonna take two arguments.
The first one is self
because we're gonna use this list of rewards here.
Because simply, we're gonna append the accumulative rewards
to this list of rewards
so we need this self to be able to get this.
So self.
And the second one is going to be the rewards
which will represent the cumulative reward.
All right, so that's our two arguments of the add function.
So now, let's go inside the function
and let's define what it has to do.
Okay, so very simply, the first thing it has to do
is whenever we get a cumulative reward, a new one,
you know, when we progress on 10 new steps,
well, what we have to do
is add this accumulative reward to the list.
And that's exactly what we're gonna do.
We're going to write a line of code
that will add this new cumulative reward
that we're getting after progressing on 10 steps
to this list of rewards here.
And to do this, we have to separate two conditions
because since we will be working with batches.
Well, the rewards will be in some lists,
but in some other cases,
the rewards can also be as a single element.
And the syntax to add an element to a list,
which is the list of rewards here, is not the same
whether you're adding a list or a single element.
So we just have to make this condition
that will separate these two cases.
And let's start with the first case
which is the case when what we're adding
to this list of rewards is a list.
And to do this, we're gonna add isinstance,
and in parenthesis, we input two arguments,
the first one is our rewards that we're adding,
so reward.
And the second one is list.
And so if isinstance, rewards, list
means if the rewards are into a list.
And so if the rewards are into a list,
what we do is very simply self dot.
We take our list of rewards
and we are going to add this list.
Because since this is a list,
what we can do is use a simple addition operation
because we can sum two lists together.
The rewards here is a list
because this will be equal to true,
I mean, if we're in this case.
And so we can simply sum this list to our list of rewards.
And therefore, we can simply add here list of rewards
plus equals rewards.
And by doing this, we're just extending the list
by summing these two lists together.
All right.
And then second condition, so we can simply add else.
So that's if the rewards is not a list,
and therefore, if it's a single element, and so else.
What happens in that case?
Well, that's the same.
We want to add the reward to our list of rewards
but we cannot use the syntax
because rewards will no longer be a list,
it'll be a single element.
And so what we need to use is another syntax,
which is the append function.
When you want to add a single element to a list,
you cannot sum the two, you have to use the append function.
And so this is exactly what we're gonna do now,
we are going to take our list of rewards of the object
and paste that here, and then add dot append.
There we go.
First one.
And of course, in parenthesis,
we input the element we want to append.
And this is, of course, rewards,
but rewards in that case, will not be a list,
it will be a single element
like a single cumulative reward, not into a list.
All right, and then we want to do this
but now, we have to add something more,
it's what does happen when this list of rewards
gets more than 100 elements?
Well, in that case, what we have to do
is delete the first element of this list of reward
to make sure that this list of rewards
always contains no more than 100 elements.
So exactly like what we did for the self-driving car
when making the score window.
And so to make sure of this,
we're gonna add a while condition,
specifying that whenever the length of our list of rewards
that is the number of elements in our list of rewards.
Whenever this number is larger than self dot size,
that is the size that we set here
in which later will be equal to 100
when we create the object.
Well, as soon as the number of elements
of this list of rewards is larger than 100,
well, what we want to do is delete the first element
of our list of rewards,
which we can get by taking the index zero.
That is the first index of our list of rewards.
This is the first element of our list of rewards
and we want to delete it whenever our list of rewards
contains more than 100 elements.
So that with this condition here,
we make sure that our list of rewards
never contains more than 100 elements.
And therefore, now what we can do
is make a new function to compute the average
of our list of rewards,
which will contain on the run 100 elements.
And therefore, we will compute the moving average
on 100 steps each time.
So let's make this new function.
It's gonna be very easy
because there is the mean function in Python,
which is a function from NumPy,
to compute the average of a list.
And so let's introduce our last function here,
which we're gonna call average.
And this function is gonna take one argument,
which is going to be self,
because we're gonna use, of course,
still our list of rewards,
which is a variable of our objects.
So self and colon.
And now, let's compute the average.
And so directly, we will return the average
because we can get it with the mean function
to which of course we are applying.
Well, what we want to compute the mean of,
that is our list of rewards.
I think I still copied it.
Yes, there we go.
So we simply return the mean of our list of rewards.
And the mean, as I said, is a function by NumPy.
So here I'm adding the shortcut,
np dot mean self list of rewards.
And there we go, we have our average on 100 steps.
Perfect.
So we made the class very efficiently.
Now we get the instructions
on how to obtain a moving average on 100 steps.
And since we're gonna use one moving average object
when doing the training,
well, let's already create this moving average object.
And so we're gonna call it MA,
and simply MA is going to be an object of the MA class.
And as we said, we want the size to be 100
because we want to compute the moving average on 100 steps.
So perfect. There we go.
We are now ready to train our AI to finally be intelligent.
It's about time.
It is from this point that our AI will become smart.
So I can't wait to train it.
It's gonna be quite easy
because this is something we already did.
But this is gonna be fun.
And besides after that, it'll be time to have even more fun
because basically, our AI will be fully ready that is built
and also intelligent.
And therefore, we will execute the code
and then our AI will play "Doom."
And eventually, we will watch the videos
of our AI playing "Doom."
And we will see if it manages to reach the best.
So I can't wait.
Let's do the training.
And until then, enjoy AI.