-: Hello, and welcome to the super exciting part
of our AI creation, the part where we make it smart.
So that's exactly what happens when
training the AI.
We will train its intelligence
to reach the goal we're wanting to accomplish.
And to do this, we're going to basically
train the neural network to output the right predictions,
and then everything is already ready
because these output signals
from the brain already have the right transmission
to the body to play the final actions.
So basically now what we're about to do is
something we already did before,
we are just going to take some random batches
from the memory, get our inputs from these samples,
get the outputs, get the targets, get the predictions,
compute the last error between the predictions
and the targets, and then perform backward propagation with
Stochastic Gradient Descent to update the weights according
to how much they contributed to this last error.
So let's do all this,
you're gonna see how it's going to be so easy
because we already have all the tools to implement this,
not only we have the PyTorch tools,
like the optimizer and the loss functions,
but also we have all the classes that we made before
like our brain, of course
which we're gonna use to get the predictions,
then our Experience Replay Implementation, Eligibility
Trace, and all these tools combined to the PyTorch tools
will make the training super performant, and therefore,
eventually we will get a super powerful AI.
So let's make this training happen.
Let's make our AI smart.
And the first thing we're gonna do now is get
the Loss Function that we'll use during the training,
when computing the error, and an optimizer.
That's the first thing we'll do, so let's create
a variable for the Loss Function.
We're gonna call it "Loss" and this will be equal to
the "MSC loss"
function from
the "nn" module.
"nn" dot "MSC loss"
That's the loss function we'll use
because basically our predictions are Q values,
you know, we're predicting the Q values
of the different actions and therefore,
since these are real numbers
well we're kind of doing some Neural Network for Regression
and therefore the Loss Function is the Means Core Error.
That's the Loss Function we use in general for regression.
All right, so now
that we have our Loss Function, let's get our Optimizer.
So "Optimizer" here, that's the variable we create for it,
"Optimizer" and we're gonna take,
as usual, as for the self-driving car, the Atom Optimizer.
That's a very powerful Optimizer that will work wonders
for the training.
So let's get this one.
"Optim" dot
"Atom"
And remember that's exactly for the self-driving car,
we have to input two essential arguments.
The first one is the one that will make the connection
between the optimizer
and the parameters of our neural network,
that is the weights of the neurons of our brain.
And to do this, we take our brain, which we called "cnn"
that's the object we created for our brain.
And so "cnn" dot "remember parameters"
There we go.
And some parenthesis.
So that makes the connection between the optimizer
and the weights of the neurons in the brain of our AI.
And then the second argument is a learning rate
and that's given by "LR"
And so here we have to take a small learning rate
because we don't want to converge too fast
and we want to have some exploration.
And therefore a good learning rate that we
can take here is a small one that is oh 0.001, that is 0.1%.
I think that's the same we used for the sale driving car.
All right, so now we have a Loss Function, an Optimizer.
So we are almost ready to start the Full Loop
where actually we will start the Full Loop right now
but just before we do it
we are gonna decide the size of the number of Epochs
we will be training the AI
and therefore I'm creating a new variable here,
which will correspond to this number of Epochs.
And let's set it equal to 100.
That will be way enough to train the AI,
and I even bet that the AI will manage to reach the best
way before 100, like 20 or 30.
Let's see, but for now, let's take 100
and if we need it, we will increase it,
but I don't think that will be necessary.
Okay, so now that we have our number of Epochs
we can start to make the Full Loop.
You know, this main Full Loop
of the training when we train over the Epochs.
So four, then the iterative variable is going to be "epoch"
That's what we choose four Epoch in.
Now of course, we're gonna use the Range Function to say
that we want to go from the first Epoch "one"
to "Number of Epochs"
"plus one" because remember
the upper bound of a range is not included.
And therefore, if we want to go up to 100,
well we have to specify in the Epochs "plus one"
to go up to 100.
All right, so "call in" and now let's get into the loop.
All right, so the first thing we're gonna do is to
do 200 runs of 10 steps.
So each Epoch will be 200 runs,
one after the other, of 10 steps.
And to do this, we have this "Run Steps" function
from our Experience Replay class, and therefore
to use this function, which is actually a method
because we will get it from our Memory object
which is an object from the Replay Memory class.
To generate these 200 runs of 10 steps,
well, we have to take our Memory Object that I
remind we created right here
"memory" is an object of the Replay Memory class
with end steps that is 10 steps and a capacity of 10,000.
We created this object, and from this object we take, well
this "Run Steps" function
"Run Steps" and we specify 200 successive runs of 10 steps.
So that will just at each epoch basically run 200 steps.
And now, now
that we have these 200 steps running at each epoch
well, it's time to sample some batches from these runs
and to sample these batches, we have another function
from our memory, which is "Sample Batch"
and that will exactly generate some batches,
from these 200 runs.
But remember, these batches are this time batches
of series of transitions that is series
of 10 steps as opposed to before
where the batches were just some batches
of single transitions Here this time there are
gonna be batches of 10 steps, 10 transitions
and therefore now it's time to get from our memory
these random batches and to get
then we use the "Simple Batch@ function to
which we have to apply the batch size.
And for the batch size where we can take 32
or even 64 or even 128,
remember for batch sizes
it's a common practice to use 32.
That's what you will see
in general in the Neural Networks Architectures
when doing some batch learning.
But this time it's quite different,
we are just sampling some batches of 10 steps.
So it's better to take batches with larger sizes.
So that's why we can take 64 or 128.
So we're gonna take 128, and actually this is gonna be
inside a four loop because we want to take several batches
and we are taking them
in what is returned by this "Simple Batch" function.
So this four-loop-four-batch in memory
Simple Batch 128
means that every 128 steps,
well, our memory will give us a batch of size 128
which will contain actually the last 128 steps
that were just run.
We're just getting some batches of size 128.
And the learning is going to happen on these batches.
And besides inside these batches
we will have eligibility trace running, you know
to learn every 10 steps.
All right, so now inside this loop, which is still happening
in one epoch, but now this time we are in a specific batch.
And so now the first thing we're gonna do is
we're gonna get our inputs and our targets separately.
And that, as I told you, it's very easy
we can do it with one
of the tools we implemented, which is eligibility traits.
As you can see here
this eligibility traits function takes a batch as input
and now we have the batch and returns as output
the inputs and the targets.
So right now
what we can simply do is create two new variables
which are gonna be the inputs and the targets.
And do this, input comma targets
equals exactly what returns this
eligibility trace function apply
to a batch.
So we will apply this function to the batch of our loop.
And so what we'll do is just
eligibility trace applied
to the batch of our full loop.
All right?
So that gets us the input and the targets.
But, in PyTorch,
there is always something more we have to do.
And of course this is to convert the inputs
of the neural network
and also the targets into some torch variables.
But no worries, there is nothing new.
We know how to do it, we can do it this way.
We take our inputs, then our targets,
and while they will be equal to variable input,
that's for the inputs,
and variable targets
and that's for the targets.
All right?
So the inputs of the brain are converted
into some torch variables, and the targets also
are converted into some torch variables.
So now we can get the input, enter the neural network,
and why do we need to do this?
That's because the next step is to get the predictions.
We have the input, we have the target,
now, of course, we need our predictions
because then what happens is that we will compute the loss
between the predictions and the targets.
So let's get these predictions to get them,
well again, this is so simple now,
we just need to take our brain, which is CNN,
our convolution neural network, and apply it to our inputs.
There we go.
The inputs go into the neural network,
and the neural network will output the predictions.
Perfect.
So now we have the predictions, we have the targets,
so we can get the loss, and that's the next step.
We're gonna introduce a new variable
because right now we're gonna get the last error
which is different than the loss function
because we use the loss function to get the loss error.
So loss error here, and that we will get it
with the loss function applied to our predictions
and the targets.
There we go.
See how everything is smooth now?
Everything is logical.
We get input first, the targets, then thanks to the inputs
we get the predictions.
And then thanks to the predictions and the targets,
we get the last error.
So, very logical and smooth.
And now what is the next step?
Well, same logical path.
Now, that we have the loss,
we can back propagate this loss error
back into the neural network
to update the weights.
And we do that with stochastic gradient descent.
And to perform stochastic gradient descent,
we need our optimizer
but we already got it here, our atom optimizer,
but now at this point, remember what we have to do,
we have to initialize it.
And to initialize it,
remember we take our optimizer object,
and then we apply
the zero grad method.
There we go,
we don't forget the parenthesis,
that initializes it.
And now next step is to back propagate the last error back
into the neural network.
And to do this while we take
our loss error
and we apply on it
the backward method.
So that's exactly to apply backward propagation.
And then finally, now that the loss error is back propagated
into the neural network,
well we can update the weight
with stochastic gradient descent.
And to do this, remember we take our optimizer
and then we apply the step method.
There we go.
The weights are now updated.
As I told you, not only we already did it,
but now it seems so simple and so natural.
So, now we're gonna do something fun.
We are going to print the average reward every epoch.
So you know, we can keep track of how the AI is going,
how the training is going.
We want to see the average reward increasing
over the steps, over the epochs.
And at first, of course there is this exploration phase.
So, the average reward might not increase at the beginning
but then once it reaches the exploitation phase,
then we'll see the average reward definitely increase.
And it will increase up to a certain level
which is when it reaches the vest as fast as possible.
So, next on with a print,
you know we are doing this in one netbox,
we have to go back to the loop here, print,
and then we're gonna print,
well first epoch, a column, then percent S,
because we're gonna convert everything
into a string that's better.
And then we're gonna add the average reward,
and then we add percent S as well.
Then we're gonna close the quote, and then we add a percent.
And on the other side, you know,
we input the variables
that are gonna be this first person S,
that is the epoch here.
And this second variable corresponding
to the average reward, which we'll compute right now.
So the average reward variable doesn't exist yet.
We're going to create it right now.
So, we are gonna use STR epoch,
even if epoch is a number
we will convert that into a string that's better.
And, we are going to add STR
that's gonna be the average reward,
and so we're gonna create a variable
that we're gonna call AVG reward.
And now we're gonna create this variable and compute it.
Okay, so let's do this.
That's the only thing we have to do left.
So, epoch we already have.
Now let's compute average reward
and we need to compute it right here
still in the epoch loop,
but out of the batch loop.
Because now we have our batch sampled,
and we have our training happening in the batch,
but now the board propagation
plus the backward propagation is done in the batch.
So we are getting back into the epoch loop
and we can now compute the cumulative rewards
which we can do with our end steps object
because our end steps object contains
this function reward steps
that allows us to get the cumulative rewards
happening in the steps, you know, during the end steps run.
So we are going to use it right now to
update the new rewards of the steps
and then we will update the moving average object
by adding the community rewards to the moving average object
and then recomputing the average.
And that's how we're gonna get the average reward.
So let's do this.
The first thing we need is the rewards that are updated.
So, let's call them rewards steps.
And then as we said,
we take our N steps object, which was I remind,
created here an object of the end step progress class
from our experience, replay foul.
So end steps object,
then we add rewards steps, and then some parenthesis.
All right, so that will get us
the new cumulative rewards of the steps.
All right?
But then we need to add these new cumulative rewards
in our moving average object.
And to do this, we have a method this time
in the moving average class, which is this ad method.
So that's very simple.
We take our moving average object
which we created here with 100 steps.
Then we're gonna use our ad method.
And then in the ad method we input our reward steps
and this will add the rewards
of the steps into the moving average.
All right, and finally,
we can compute the average reward.
And that is, well, you know, that's the same variable here.
So that's what is going to be equal to the average reward.
And to get it
we just need to use the average method this time
for more moving average object.
And that is we do NA.average.
Just like that because our moving average
was already updated
with the new reward steps that we just added
thanks to the ad method.
Great, so now we have our average reward.
So that will populate here
and this is going to be printed every epoch.
All right, so we're done.
So, I'm so excited to see the results
and actually
I'm gonna have a surprise for you
in the next tutorial while watching the results.
So, it's gonna be pretty exciting.
And so now I guess it's time to play with the AI
and have fun.
All right, so prepare yourself a good coffee or a good tea.
Now it's time to sit comfortably in our chair
and watch some very cool videos of our AI playing Zoom.
So let's do that to the next tutorial.
And until then, enjoy AI.