-: Hello and welcome to this Python tutorial.
All right, so we just updated the memory
after reaching the new state
and now let's take care of the next update.
According to you now,
what is going to be the next update?
Well, basically we are done with one transition.
We updated the last element of the transition
which is the new state.
So now it's like we're starting all over again.
And when we are starting all over again, what it's like
you know, we are in this new state at the environment
and so what do we need to do now naturally?
Well, of course it's to play an action.
Because we already got the observation of the new state.
So now, next thing that we have to do is play an action.
And therefore what we need to do now is of course
use the select action function to play the action.
So let's do it.
Let's create a new verbal action
and let's play the action with the select action function.
So I'm taking while first self to specify
that the select action function is a method
of the object of the De Qing class that will be created.
So self dot select action. Here we go. Self dot
select action. And then of course,
since the select action function takes the state as input
because of course the select action function
will return the output of the neural network when
the current input state entered the neural network.
So we have to input the input state here
and since that's the state that we just reached
in the environment right now, where the input state is
of course new state, because the state that we just reached
at the time we were right now is new state.
So in the select action function
I've been putting new states.
All right, so with this line of code
we simply play the new action after reaching the new state.
Okay? And now that we played in action, well
we get the reward and therefore we
get a feedback with the reward.
And therefore, if we have more than 100 elements
in the memory, well it would be time to learn.
And therefore, what we must do now is what logically comes
after selecting an action, which is of course to learn.
The AI needs to start learning
if it is doing the things the right way.
And now since it just played the action, well
we're gonna make the AI learn
from its actions in the last 100 events.
But before we apply this learn function
we have to make this if condition to make sure
that we already have reached more than 100 events.
Because you know, we're learning
from the random symbols of the memory.
You know, we have this huge memory of 10,000 elements.
We're taking some random samples
of the memory of 100 elements
and the AI is learning from the information contained
in this sample of 100 random transitions.
So let's just make this if condition
to make sure that the number
of elements of the memory, self dot memory,
and then be careful. Just a little trick here,
self dot memory is the object of your replay memory class.
But then the replay memory class has a attribute
which is memory.
So in fact, we need to take self dot memory dot memory.
The first memory is the object of the replay memory class.
And the second memory is the attribute here,
self dot memory.
So, if the number of elements in the memory is, well
we want it to be larger than 100, then Colin
and then what happens?
Well, we can learn, but before learning
we need to get this random sample of 100 transitions.
And this we can get with the simple function.
And since the simple function returns the different batches
the states at time T, the states at time T plus one
the actions at time T and rewards at time T.
Well, what we need to do now is create some new valves
which are gonna be the batch of the states at time T,
the batch of the next states,
the batch of the rewards and the batch of the actions.
And we can simply give the same names
as we gave for the arguments here and base that here.
And these variables will be equal
to what the simple function returns
because it returns exactly these batches of the states,
next states, rewards and actions.
So what we simply need to do now is get first
our memory object, and then from this memory object
we are gonna use the simple method
which will take as input.
Well, the number of transitions we want our
AI to learn from, that is 100.
That's why we made sure
that the memory had more than 100 transitions.
So it's gonna learn from 100 transitions of the memory
so the learning will be much better.
And so now, let's make this learning happen.
Well, since the learn method is a method
of our De Qing class, well we need to
access this learn method from the future
objects that will be created from the De Qing class,
and therefore what we need to take is self.
Self refers to that object to the De Qing class
and then learn as this learn method.
Learn method to which we input of course, these guys here,
the batch state, the batch next state, the batch reward
and the batch action.
These are our batches, sampled from our memory
and we get 100 of them because we have 100 transitions
and from this 100 transitions, we take 100 states
100 next states, 100 rewards, and 100 actions.
So let's face that here.
And there we go. Now the learning will happen.
It will happen from all these random batches.
Perfect. And now what we need to do is the very
last updates after, you know
reaching a new state and playing in action.
Well, we got the action to play
but we still didn't update the action that is ourselves
that last action variable.
So let's make sure we don't forget this.
Let's do it right now.
We will update the last action self dot last action equals
and of course, action.
The action that we just pay here
with the select action function.
All right, so now the last action is updated, then same
for the new state. We reached the new state
but we haven't updated the last date yet, because
of course the last date was before the state at time T.
But since now we reached the new state T
plus one at time T plus one.
Well, the last state becomes this new state here
and therefore we need to update it as well.
Self dot last state equals our new state.
There we go. And now what do we need to update?
Well, there is only one thing left.
That's of course the reward.
And the reward is exactly the reward we get in reality.
That will be the argument of this update function
which if we make the connection to our map,
will be the last reward. That is the reward we get right
after playing the action in this reached new state.
So if we go onto some sand
this last reward will be bad minus one.
If we get further from the goal
we will get a slightly bad reward minus 0.2.
If we get closer to the goal
we will get a slightly good reward, open one.
And if we get too close to one edge of the map
well that will be a bad punishment.
We will get minus one for each edge.
So that's the last reward we get.
In reality, that is the one that happens
for real on the map.
And this will be the argument of the update function.
This last reward here, that's exactly this one.
And so, since this is the argument of the update function
while that corresponds to this reward here
and therefore our self dot last reward variable initialized
at the beginning in this it function
becomes the new reward we get.
In reality that is reward, or that's the same last reward.
All right, so now we updated our last reward.
And now since we just got our last reward,
well we can now update the reward window.
You remember? The reward window we initialized here
as one of the variable of the object of our De Qing class.
That's the window that's going to keep track
of how this training is going
by taking the average of the last 100 rewards.
So you know, it'll be like a sliding window
showing us how the mean of the reward is evolving.
And so since we just got our last reward
well we can update the reward window.
And so how do we update it?
Well, we simply need to append this last reward
to the window.
And therefore what I'm gonna do is take my reward window
self dot reward window. Here it is.
And then I'm going to use the append function.
And inside the append function, we need to
input the element we want to append to the reward window.
And that's of course the reward.
All right, perfect.
And then, since this reward window
is going to have a fixed size
you know it's not going to be a growing window
it's going to be a window, a fixed size sliding
with time to show us the evolution of the reward.
And so now we need to decide for a size of this window
and it's simply the number of means
of the rewards we will have in this window.
And so for example, let's get, you know
the last 1,000 means of the last 100 rewards.
And so to make sure of it,
we're gonna add if then len, then we'd say
our reward window and we simply add here, if the number
of elements in the reward window is larger than 1,000
well what we wanna do is delete the first element
of this reward window.
And the first element of this reward window
has the index zero.
All right?
And now we make sure
that this reward window will never get more
than 1,000 elements.
That is 1,000 means of the last 100 rewards.
So that's perfect. This will be a window,
a fixed size, so that we can see if the mean
of the reward is increasing and therefore if the training
is going well and accordingly the car does what we want.
Perfect. And now one tiny little thing to do left according
to you, what is it going to be?
Well, remember this update function not only
updates the different elements of the transition
and the reward window, but also it returns the
action that was played when reaching this new state.
That's why we have in the map action equals brain of update
last word, last signal
and therefore it's supposed to return something
and there's something it is supposed to return is
of course the action.
So the simple last thing we need to do here,
is just return action.
The action that was displayed when reaching the new state
and that's it.
Our update function is ready
It's going to do all the required updates
and it'll return the action when reaching the new state.
That's perfect.
That was the last difficult action to make
for all this AI process.
Now the rest will be kit stuff.
We will just make a score function to return the means
of the rewards in the reward window,
then we will make a safe function to save the brain
of the car whenever you want to quit the application
and go back to it and of course,
since you want to be able to load the brain of your car,
when you get back to it, get back to the application.
Well, we will end up by making a load function
which will load your model
after you saved your model with the safe function.
So three functions to do left, but it's going to be simple.
And then we'll have the most exciting section
of this first module. That is the demo.
We will see if the AI works,
we will see if the car reaches the goals
and we will see how we can improve it.
And then eventually you will have built your first AI.
So I can't wait to start the demo.
Let's make these three functions first and until then.
Enjoy AI.