Narrator: Hello and welcome to this Python tutorial.
All right, so now in the next function
that we're about to implement,
we will train the deep neural network
that is inside our artificial intelligence.
So basically we're gonna do the whole process
of forward propagation and then back propagation.
So that it is, we're gonna get our output,
we're gonna get the target,
we will compare the output to the target
to compute the loss error.
Then we're gonna back propagate this loss error
into the neural network
and using stochastic gradient descent,
we will update the weight
according to how much they contributed to the loss error.
So let's do all this.
For those of you coming from the deep learning course,
this will be good stuff,
but for the others, don't worry,
I'm going to explain that again.
So we're gonna call this new function learn.
And this learn function is going to take several arguments.
First self, of course,
which will refer to the object of the DQN class.
Then we're gonna take our batch state
for the current state,
then our batch next state
then our batch reward,
and finally our batch action.
So why do we take this?
You probably recognized what is the series.
Well, that's of course a transition.
A transition of the mark of decision process that is
at the base of deep Q learning.
And why do we all take them into some batches?
Well, that's because, you know, remember,
We don't consider the transitions
by a series of the top current state,
next state, current reward, and current action.
We created some sample batches here
thanks to the sample function.
And so now our transitions are in the form,
the first batch for this date,
a second batch for the next date,
a batch for the reward, and a batch for the action.
That's the form of our transitions now
and they're all well aligned with respect to time
thanks to this concatenation that we made here
with respect to the first dimension.
So the point is, now we have these transition of batches.
One batch for each of the states,
next state, reward and action.
And we do all this
because we're using this experience replay trick
so that our deep neural network can learn something.
Remember, if we only had the transitions by themselves,
well it would be some instant learning
or if you want some very short memory learning
and therefore the model wouldn't learn anything.
So we have to take these batches from the memory
which become our transitions
and then eventually we will get the different outputs
for each of the states of the input batch state.
And we will do this for the batch state
and for the batch next state
because we will need both to compute the loss.
I will soon remind the balance equation
that is at the heart of the deep Q learning algorithm.
So now let's go into the function
and let's first get the outputs of the batch state.
So I'm gonna call this first variable outputs.
And then we're gonna take of course our self dot model.
So self dot model,
because we want to get our model outputs
of the input states of the batch state.
And since our model is actually expecting a batch
of input state
well we can totally input batch state right now
for the input of the model,
that's exactly how we initialized this state
that are going into the network
with a torch tensor, with this fake dimension for the batch.
So that's perfect.
We now get the outputs of the model.
But then there is another technical trick.
If we only do self dot model batch state
well we will get the outputs
of all the possible actions,
you know, zero, one, and two.
But that's not what we want.
We are only interested in the actions that were chosen.
The actions that we're decided
by the network to play at each time.
And so to get these action we're interested in,
that is the action displayed,
well we have to use this gather function
in which we input one,
because we only want the action that was chosen
and then we add batch action.
With this, the one and the batch action,
we will gather each time the best action to play
for each of the input states of the batch state.
We only want the action that is played
the action that is chosen.
And we get this with this gather one and batch action.
But then be careful,
the batch state here has this fake dimension
corresponding to the batch,
and batch action doesn't have it.
Batch state has it because we used the unsqueeze here,
but we haven't used any unsqueeze for the actions.
So we have to add it here
so that the batch action has the exact same dimension
as the batch state.
So we're gonna add a dot unsqueeze zero right here.
And actually this is not zero, but one,
because zero corresponds to the fake dimension
of the state
and one will correspond
to the fake dimension of the actions.
And finally, the last thing we need to do here
is we need to kill this fake batch with a squeeze.
And why do we need to do that?
Because now we are out of the neural network,
we have our outputs, but we don't want them in a batch.
We want them in a simple tensor, or a simple vector
a vector of outputs.
The batch is just when we work in the neural network
because the neural network is expecting the format
of tensors into a batch.
But now we have our outputs
and in the next balance equation of the deep Q learning,
we won't need them into a batch.
So I'm killing the batch here.
I'm killing the fake dimension
to get back the simple form of our outputs.
So I'm just adding here dot and then squeeze.
And then, since I want to kill the fake dimension
corresponding to the batch of the action,
well since this fake dimension has index one
I'm adding one here.
All right, and now there we go,
we have our outputs.
Okay, we have a little warning, what is it?
Local variable outputs is assigned but never used.
That's okay, we will use it very quickly.
So that's our outputs.
And now we want to get our next outputs.
So now you might be thinking
why do we need the next outputs?
Well, to understand this, we need to go back
to the deep Q learning algorithm, which is right here.
That's a part of the Lattik handbook.
So that's the whole deep Q learning process.
At the beginning, we're initializing all the Q values.
And then at each time T, well, there we go,
we select the action with Softmax.
That's what we did with the select action function.
Then we append the transition and then,
as you can see, we get the prediction,
we get the target and we compute the loss.
So why do we need the next outputs as well?
That's because of the target.
The target is equal
to gamma times the next output plus the reward
And we will compute the targets right after that.
But since we need the next output for the target,
let's compute this first.
So again, to get the next output, that's very simple.
The next output is going to be the result
of our neural network
when the batched next date is entering it as input.
So very simply, we take our model,
that is our neural network
and then this time the input of the neural network
is going to be the batch next date.
The batch next date.
But now remember,
if we go back to the deep Q learning algorithm,
well you can see that the next output
is the maximum of the Q values for the next state
with respect to all the actions.
So right now, to get the next output
we need to get the maximum of this Q values.
And therefore I'm gonna do here a detach,
you know to detach all the outputs of the model.
Because we have several states in this batch next states.
That's the batch of all the next states
and all the transitions taken
from the random sample of our memory.
So I'm detaching all of them using the detach function.
And then I'm taking the max of all these Q values.
And since we are taking the maximum of these Q values
with respect to the action,
well we have to specify
that it is with respect to the action.
And since the action is represented by the index one
well again, we have to put the index one here
and then we have to specify that we're taking the Q values
of S.T. plus one, that is the next state.
And the next state is represented by the index zero
because the index zero corresponds to the state.
And therefore here we need to add brackets
with the index zero.
That way we get the maximum of the Q values
of the next state, represented by the index zero
according to all the actions
that are represented by the index one.
And now, perfect, we get our next outputs.
These are unused yet,
that's why we had the warning,
but that's fine.
We will use it right now to compute the target.
And speaking of the target
that's the next step of this learn function.
So there we go,
target equals...
Now let's get back to our AI handbook.
The target is equal to the reward
plus gamma times the next output,
that is the maximum of the Q values of the next date
according to the actions.
So there we go, let's compute that.
So that is equal to self dot gamma.
And self dot gamma was initialized here.
It is introduced, so that's a variable of our DQN object.
Self dot gamma times the next output,
as we just said, plus the reward.
That is, the batch reward.
We're working with batches here.
So plus batch reward.
And that's the targets in one sample of the memory.
Gamma multiplied by the next outputs plus the reward.
All right, perfect.
So now we have our outputs, we also have our target
and therefore we can compute the loss.
The loss that is representing the error of the prediction.
So let's call this loss TD loss.
TD is of course for the temporal difference.
That is again at the hearts of Q learning.
And this TD loss is going to be equal to the Huber loss.
That improves much the Q learning.
That's the loss function we will choose
for our artificial intelligence.
For those of you coming from the deep learning course
that's really the loss I recommend
if you want to implement deep Q learning.
And so how are we gonna get this Huber loss?
Well, again, we're gonna take a function
from the functional module, represented by F,
and therefore here I'm going to use
our functional module F dot
and the Huber loss can be obtained
thanks to the function smooth L one loss.
That one. So pressing enter.
And that's really the best loss function I recommend
for deep Q learning.
It really improves DQ learning,
but this is a function, so I'm adding some parenthesis
and now there is nothing more simple.
The arguments we need to input
are the predictions and the targets.
So the predictions is of course our outputs,
because that's the output of the neural network.
You know, the output of the neural network
is what the neural network predicts.
So that's the prediction.
So the first argument here is outputs.
And then the second argument is of course the target,
the thing we're trying to get,
and it's already computed.
Perfect, we can directly input target.
Perfect. And now we have the loss.
Just forgot a little T here.
There we go, now the warning should disappear.
Yes, perfect.
And now that we have the the loss error
we can back propagate this error
back into the network to update the weights
with stochastic gradient descent.
And that's exactly what we're gonna do in the next step.
So of course, now what we have to do, as you might guess,
is take our optimizer,
our optimizer which, again we introduced here.
We initialized it.
And that's an Adam optimizer
which is actually an object of the Adam class.
And it is already fitted with the parameters of our model.
And we already chose a learning rate of 0.1%.
So perfect, our optimizer is ready
but now we need to apply it on the loss error
to perform stochastic gradient descent
and update the weight.
So when working with pytorch
the first thing we need to do is reinitialize it
at each iteration of the loop.
We must reinitialize the optimizer
from one iteration to the other
in the loop of the stochastic gradient descent.
And to reinitialize it at each iteration of the loop
well we're gonna use the following method
which is zero grad, here we go.
Zero grad will re initialize your optimizer
at each iteration of the loop.
Then let's not forget the parenthesis.
Perfect. And now that it is reinitialized,
well we can perform backward propagation with our optimizer.
And how do we do that?
Well we take our TD loss
and we are going to back propagate it back into the network.
And to back propagate it into the network
we need to use the backward function.
And inside this backward function,
I recommend to input retain underscore variables
and set it equal to true.
I recommend to do this
because this will improve back propagation.
The use of written variables equals true
is to free some memory and we need to free the memory
because we're gonna go several times on the loss.
So that will definitely improve the training performance.
And finally, last step of this learn function
is to update the weights according
to the back propagation, that is according to how much
the weights contributed to the error.
And to do this, we take our optimizer again,
which was initialized and reinitialized
and we use the step function.
And simply with this line of code
by using this step function, this will update the weights.
That's this line of code that updates the weights.
This line of code back propagates
the error into the neural network.
And this line of code uses the optimizer
to update the weights.
And there we go.
We have a learning neural network.
All right, so congratulations.
This was probably the most technical and difficult part
of all this deep Q learning model.
I know by Pytorch can be tricky sometimes
with these unsqueeze and squeeze,
but in the end I promise you will get
a very functional neural network and
therefore deep Q learning model
and eventually a great artificial intelligence.
So now let's move on to the next step
of our deep Q learning model,
which will be the update function
that will obviously update
when the AI will discover the new state.
So you know, it will discover the new state
and then it will receive the reward
depending on the action that it displayed
and this new state.
So we'll take care of this with the update function
and we'll do this in the next tutorial.
Until then, enjoy AI.