-: Hello and welcome to this Python tutorial.
All right, so today, we will be making the update function,
which will update everything there is to update
as soon as the AI reaches a new state.
So when it reaches a new state,
you know we need to update the action,
the last action becomes the new action that was displayed,
but also, the last state that becomes the new state,
and finally, the last reward
that becomes the new reword we get when we play the action.
So that's the logical path
that happens right after selecting an action.
We need to update all the elements of the transitions.
And of course, we will get a new transition,
so we will have to append this new transition to the memory.
And finally, we will also update our reward window
to keep an eye on the evolution of how the training's going
and how the exploration is going.
But what's most important for you to understand
is that now we can finally make a connection
between the AI that we're implementing right now to our map.
Because, if we go back to our map, remember,
there is this big update function into the game class,
so that's where we're actually making the game with the car
and defining how the car should be punished
when it's making a mistake.
But in this game class, we notice this update function.
And in this update function, we notice this line,
action=brain.update, last reward, last signal.
And actually, this is exactly what we're about to make.
We are about to make this update function
that will take the last reward and the last signal
to get the next action to play.
So not only will we update
all the different elements of the transition, but mostly,
we will be playing the action that we should play
when getting the last reward and the last signal.
And so, of course, in this update function,
we will use the Select Action function
that we just implemented before.
We will integrate the Select Action function
in the future update function that we're about to make
to select the right action to play
besides making all the updates.
So that's really important to make this connection
with the map right now.
What we're about to make is, eventually,
the connection between our AI and the game,
the game that we make in this class.
And so, what we can do now is directly take this update,
last reward, last signal,
because that's exactly the function that we will be making
with these two arguments here.
And just as a quick reminder, brain is our AI object,
that is, it's the object of the DQN class.
So what we're gonna do now is we are going to copy this,
update, last reward, last signal,
and that's gonna be our next function we're making.
And therefore, I'm pasting that here.
Then, just to be careful, I would just like
to give some different names than the names we have here.
We have last reward here
and I don't wanna confuse this last reward with this one.
That can be dangerous.
So I'm going to replace last reward here by reward
and by doing the same for last signal.
Let's just put signal or even new signal to specify
that we want to make the update when reaching a new state,
and therefore, getting a new signal.
But then, of course, this reward here
is going to be the last reward that we get here.
When going on to some sand or worse,
getting too close to an edge of the map.
That's where we define the last reward.
This last reward is going to be the input
of the update function,
so that's why we have last reward here.
But right here, I'm just giving another name
for the argument reward
to not confuse it with last reward here.
All right, so this is the update function.
And now, let's go inside it and let's do these two things,
that is update all the elements of our transition
and, of course, select the action.
Okay, so what do we need to update first?
Well, as you understood, we want to make the updates
when reaching a new state.
So the first thing we'll be updating is,
obviously, this new state.
That is the new state we're reaching.
So I'm gonna call this new state new state and then equal.
And so, how can we get this new state?
Well, of course, that depends on the signal.
The new signal that the sensors just detected.
And as a reminder, the state is the signal itself
composed of the three signals of the sensors,
signal one, signal two and signal three,
plus orientation and minus orientation.
That's our state, so be sure to understand
that the signal is the state.
But right now, it is a simple list of five elements.
And since this is going to be the input
of the neural network, remember,
we have to convert it into a torch tensor.
So that's exactly what we're gonna do right now.
We are going to take our torch library
and then take the tensor class, there we go,
which will convert our new signal into a torch tensor.
Then, it's better to make sure
that all the elements of the torch tensor are floats.
So I'm going to make a type conversion
to convert them into float, like this.
And then, finally, try to get the reflex
of what we need to do next,
it's, of course, to create that fake dimension
to add the dimension corresponding to the batch.
And we do this, of course, with the Unsqueeze function
to which we have to input the index of this fake dimension
we want to have for the batch, which is zero.
All right, and now we have our new state
composed of the three signals of the three sensors,
plus orientation, minus orientation.
And of course, that will depend on the new signal
we are getting with this update function right at this time.
Last signal, we get the three signals,
plus orientation, minus orientation.
And as a reminder, the three signals
are the density of sand detected around the sensors.
All right, so we just got our new state,
so that means we reached the new state.
And now, we have to make the next update.
So according to you, what do we need to update now?
What would be the logical thing to update right now,
after reaching this new state?
Well, what we need to update now is the memory.
Why is that?
It's because, at each time T,
a transition is composed of the current state, ST,
the next state, ST + 1, the reward, RT, and the action, AT.
And right now, we already have ST,
we already have RT and we already have AT.
And we just got the last element of the transition, ST + 1.
So by getting this new state, ST + 1,
we are getting one brand new transition of the memory.
And therefore, we have to append this brand new transition
to the memory because that's simply our next transition.
So that's why we have to update the memory right now.
And therefore, what I'm gonna do is take my memory object
created from the replay memory class,
and therefore, I'm gonna take self.memory
to refer to the object.
But since I'm using self,
I have to include the self in the update function.
So now, you can really see what the self is for.
It's whenever you use one variable
that you created and initialized in the init function.
So self.memory, and now we need to update it.
And, according to you, how are we going to update that?
Well, the good news is
that we already made a function to do that.
It's the Push function,
which appends an event or a transition to the memory.
So that's exactly what we're gonna use now.
We're gonna use the Push function
to append our new transition that we just made
to the memory.
And therefore, here, I'm taking not an equal
because we're gonna use a method,
and therefore, we can directly use .push.
And first, I'm going to add the transition,
this new transition that we just got,
and that is first, the last state.
So self.laststate.
So that's ST.
That's exactly this one, it already exists.
Then, the next element of this transition is, of course,
the new state that we just reached.
And therefore, since it is not a variable of the object
that we created and initialized in this init function,
we don't put a self here.
We directly put the new state.
Then, the next element of the transition is the action.
And say, we already have the last action,
which is this self.lastaction here.
So, of course, it is equal to zero, but then, of course,
it will be updated with the Select Action function.
But that's this one, so then it is self.lastaction.
But now, be careful.
The elements that we're including in this transition
should all be torch tensors.
As you can see, that's the case for the last state.
It's a torch tensor.
The new state is also a torch tensor.
And so, this must be the same for the action
and then the reward, of course.
But now, you're gonna think, "How can it be a torch tensor,
"considering that it's simply a number?"
The action is either zero, one or two.
But, in fact, that's not a problem.
We can still convert this zero, one or two variable
into a torch tensor.
This will just be what we call a long tensor.
The long is a type and that's the tensor
that will contain an integer
because the last action is an integer.
It is zero, one or two.
So what we're gonna take now is our library torch.
Then, we're gonna take the long,
here it is, the long tensor class.
That will create an object,
which will be the long tensor itself.
And by taking this self.lastaction function as input,
it will create this long tensor object,
but it will still contain zero, one or two
into a long tensor object.
And that is just to be consistent with the transition
that should only contain tensors
because we're working with PyTorch
and we're working with a neural network,
so we have to work with tensors.
So there we go, torch, long tensor
and one last conversion to make.
We must be sure that what's inside this long tensor
is an integer.
And to make sure of it, even if we already know
that the last action is zero, one or two,
to make sure of it,
we're gonna make this int type conversion again.
We convert our self.lastaction into an integer.
There we go.
And then we must just put the integer self.lastaction
into brackets, right here,
so that now we get a long tensor of one element,
which will be this last action zero, one or two itself.
So the key point is that's just how you convert
a simple number, zero, one or two into a tensor with torch.
All right, and then, finally,
the last element of the transition.
And that's, of course, the last reward we got.
So that's exactly the last reward variable we created
in the init function that was initialized to zero,
but then, of course, is updated right here in this code,
either when we go into some sand,
which is a negative reward,
or if we get further away from the goal,
that's again, a negative reward.
If we get closer to the goal, that's a positive reward.
And the worst punishment,
if we get too close to one edge of the map,
well, that's a terrible negative reward, minus one,
and that's all.
So let's add this last element of the transition,
self.reward.
So I'm copying this.
Pasting it here.
And now, we have to make another conversion,
which will be, of course, exactly the same as this one.
Only, since the reward is not an integer,
but a float number,
we will simply make a torch.tensor conversion,
but without the int.
We will keep the brackets here because,
first, we have to put the number into a list
and then this list will go as input
in the torch tensor class.
But we don't have to make that int conversion
because last reward is a float number.
So what we're gonna do is simply add here torch.tensor.
Torch.tensor, then parenthesis, brackets,
and we're gonna close the brackets here,
and we close the parenthesis.
There we go.
So to summarize, with the new state
that we just reached and the reward,
we observe a new event of transition
that we add to the memory.
And this transition contains the last state, ST,
the new state, ST + 1, the last action played, AT,
and the last reward, RT.
All right, and now, we are good with our memory update.
So let's have a quick break
and we will take care of the next update
in the next tutorial.
Until then, enjoy AI.