-: Hello and welcome to this Python tutorial.
So let's do this.
Let's make this full loop,
starting from the right and going to the left.
And to do this, we're gonna add four.
So this time the iterative variable is going to be our step
because we're gonna go from the last step
to the first step of the series of transitions.
And so four step and on the trick
to go from the right to the left
is to use for step in, reversed.
Reversed, and now we just need to input a sequence.
And this sequence is going to be of course, our series.
So we input our series, but as you can see in the paper
we go from T minus one to T start.
So we don't go from the last last step
that is the terminal state but the state before that,
that is T minus one,
but to T start, that is the first step.
And so here to go from
not the last state but the state before,
we need to add in brackets column minus one.
I'm sure that for those of you
who followed the machine learning
and the deep learning course you know this trick,
column minus one means that you're going up to
the element before the last element
but not up to the last element.
And therefore we get the sequence we want.
That is we're gonna go from the element
before the last element up to the first element
and that we do things to reversed.
To go from the right to the left.
All right, so we are ready to enter the full loop.
And so inside this full loop, what are we going to do?
Well, we're gonna do exactly as in the paper.
We are going to update the accumulative reward
by multiplying it by Gemma
and adding the reward obtained in the current step.
That is in the step of the full loop.
All right, so let's do this.
Going back to Python.
And so we want to update our accumulative reward
the following way by first multiplying it by Gemma.
There we go.
Here, we multiply it by Gemma.
And then we want to add the reward of the step
which we can access this way with this special structure.
Remember that reward is a attribute of the step object.
And so here of course, we add a plus.
So accumulative reward equals the reward of this step
we are in right now, the loop,
plus Gemma times the previous cumulative reward
before it is updated.
Perfect, so now I think we're good.
We're following thoroughly the algorithm.
And now time for the next steps.
Well, now it's going to become pretty easy.
We go back to the first full loop
because this full loop is just to
compute the cumulative reward,
going from the right to the left
by updating this way, following the algorithm.
And now as you remember, the goal of doing all this
is to get our inputs ready and our targets ready
so that we can minimize the square difference
between the two for the training.
And so right now the only thing that we have to do left
is get these inputs and targets ready.
So let's do this.
First, what we need to do is add
the first state of the series in our inputs list.
So far this input state is in this input variable
but that was just to compute the output.
So we're gonna get this input state
of the first step separately
because that's exactly what we need to append
in our inputs list.
So let's get this separately
therefore we're gonna call it state.
And so exactly the same as here,
we can get it this way
by taking the first index of the series
which contains the first transition,
and then adding that state
to get the state of this first transition.
So that's the state we need.
Then same, we're gonna get separately the target
associated to this input state of the first transition.
And so I'm introducing a new variable here, target,
which will be equal to the Q value of the first step.
And since the Q value is returned by the new network
it is contained in output.
And since output is the output associated to this input
which contains the first element of the transition
while we can get this Q value of the first state
by just taking output here and taking the index zero,
and then we add dot data.
That will simply get us the Q value of the input state
of the first transition.
And that is exactly the target Q value.
So that's why we're taking it.
Then we are going to update this target variable
but only for the action that was selected
in the first step of the series.
And to access this first step of the series
well, we need to take first series zero
because this is exactly
the first step of the series, series zero.
And to access the action corresponding to this first step
of the series, well, we need to add here, dot action.
Again, that is this attribute structure that we're using.
You know, action is a attribute
of the first step of the series,
that is the first transition of the series
because each transition of the series
has the following structure;
state, action, reward, and done.
So action here, this attribute action here,
means that we're simply getting
the action of this first state.
And so the target for that specific action
of the first step is exactly what needs to be updated
by the cumulative reward.
So basically here we're just gonna write that
the target associated to the action that was played
in the first step of the series is this cumulative reward
that we just computed.
All right, and now we're finally ready to update our input
by appending this first input state here,
and this first target here for the first step.
We only need to update the first step of the series
because you know, we train the AI on 10 steps
and therefore the input is the first step of the 10 steps.
And also we get the target in this first step.
But then we don't need to get any inputs
or any targets in the following steps of the 10 steps
because basically the learning happens 10 steps afterwards.
So that's why right now we are only getting the state
and the target of the first step of the series.
So it's important to understand that.
And therefore, if we understand that
then now we understand that we have to input them
in our list of inputs and our list of targets.
So let's do this.
First, let's append the states to our input.
So we take our inputs list
and we use the append function to add the state
which remember is the input state
of the first step of the series.
And then we are going to append the target
of the first step to our list of targets.
And to do this, we take our list of targets.
And same, we use the append function
to append this first target.
There we go, almost done.
And now we need to return the last things
which are of course what we need
that's what we said at the beginning of this tutorial.
The inputs and the targets that are now updated.
So we're gonna add here return
then we're gonna get our inputs first
but then that's the same we need to convert them
into a npi array first
then do a type conversion to make sure we have
a single type with D type equals
np.float32 the same.
And then we convert this into a torch sensor
because of course we're working with pi torch
so that's totally compulsory.
And so I'm using the torch from npi function again,
and that gives us our inputs.
Perfect, and now let's do the same for the targets.
Now we're gonna use this trick, which is quicker.
We're gonna stack the targets together.
And to do this, we need to take first our torch library
because we're gonna use the stack function
by torch to stack the targets.
And so this line of code basically returns the inputs
and the targets that were just updated
through this eligibility trace (indistinct) algorithm,
or we can call it N-step Q learning.
And so now congratulations,
we are ready to do the final training
because basically the training consists
of minimizing the square differences
between the predictions of our inputs and the targets.
So let's get our AI smart.
It will become smart in the next tutorial.
And so until then, enjoy AI.