-: Hello and welcome to this Python tutorial.
In this tutorial, we're gonna make the first step
into implementing the Deep Q-Learning model.
So basically, we're about to implement
the whole process of the Deep Q-Learning algorithm.
And so we are gonna use what we created before
that is the architecture of the neural network
to replay memory,
to integrate this into the whole Deep Q-Learning process.
And this whole Deep Q-Learning algorithm
is going to fit into one class.
So, that's the last task we're making
to implement artificial intelligence.
And this class will just contain different functions.
So, we will have the init functions
which will create and initialize all the variables attached
to our future Deep Q-Learning objects
which will represent the Deep Q-Learning model itself.
And then, we'll have some other functions.
One of them will, of course,
be to select the right action at each time.
We will also have an update function,
a score function to get the score
and have an idea of how the learning is going,
if it's going well,
if the exploration is going well
and if it can move on to exploitation.
And then we'll have a save function to save the model,
that is to save the brain of the car,
and then eventually, a load function.
So, we have a couple of functions to make.
We're gonna make one function for each tutorial.
And today, we're gonna start with the init function
as usual when we're making a class.
But first, let's not forget to introduce the class.
So, we're gonna call it DQN for Deep Q-Network,
then some parenthesis, colon,
and there we go with our first function.
So, let's do this.
Def then double underscore, init,
double underscore again and parenthesis.
So as you understood in this init function,
we are going to introduce the variables
attached to our objects.
So, we're gonna have a couple of lines starting all by self.
And we will basically create and initialize
all the variables that are needed
to implement a Deep Q-Network.
So we will, for example, create an object of our network
because of course, we need our deep neural network.
Then we will need our memory,
we will create another variable for the memory.
So we'll have another variable, self dot memory.
But then that's not all, we will have to create as well
some variables for the last state,
the last action and the last reward.
That's of course, you know, the variables
that you see in the Deep Q-Learning algorithm.
And then, what else?
Well, we will also need an optimizer, you know,
to perform stochastic gradient descent
to update the weights
according to how much they will contribute to the error
when the AI is making a mistake.
And then, I think that's all.
That's basically, the variables
we now need to create and initialize.
But in this init function,
we will input a couple of arguments.
First, as usual, self,
which is the argument referring to our object,
then since, you know, we're going to create
an object of the network class,
well, since the network class
takes as argument in the init function,
input size and ambi action.
While that's the same here,
when creating an object of the network class,
we will need to choose an input size argument
and an ambi action argument.
Therefore, we can just copy them
and paste them here and here we go.
So, these arguments will now become
also some arguments of the DQN class.
Whenever we create some future objects of the DQN class
that is some future Deep Q-Learning models,
well, we will need to specify the input size
which I remind is the number of dimensions in the vectors
that are encoding your states, your input states,
and a number of actions
which is the number of possible actions the car can make.
So I remind these are, either go left,
go straight or go right.
Okay, perfect.
Then you know, we will be creating
an object of the replay memory class
to create the memory object
to get our memory of the transitions.
And in the init function, we have the capacity argument.
But since we will only be using it once,
actually, when we create the memory
and not anywhere after,
well, we won't need to specify a capacity argument.
We could do this,
but we will directly input the number of transitions
we want our memory to have.
But then, we need one last argument
which is the gamma parameter in the Deep Q-Learning model.
Remember, this gamma parameter is the delay coefficient.
That's a parameter of the equation
and therefore we will put it here
because we will be using it afterwards several times.
So, let's put it here.
We are gonna call it gamma
so far, that is just the name of the argument.
And there we go, that's all the arguments we will need
for this init function.
So that means,
that whenever we create our Deep Q-Learning model,
that is whenever we create an object of the DQN class
well, we will need to specify as argument,
the input size, the number of action
and the gamma parameter
and we'll input the real values for them soon.
All right, so now let's go inside the init function.
Okay, so now basically this is going to be easy.
We are just about to create and initialize
all the variables that we'll need.
And so, let's start with the first one
let's start with gamma, actually, the delay coefficient.
So since this is the variable
we want to be attached to our object,
we'll start with self
so gamma is going to be a variable of our DQN model.
So self dot gamma equals the argument that will be input
when creating an object of the DQN class.
So gamma, and there we go with the second argument.
So the second argument is going to be the reward window.
So what is this reward window?
Well, that's gonna be the sliding window
of the mean of the last 100 rewards,
which we'll use just to evaluate
the evolution of the AI performance.
You know, we'll have the mean of the reward
into this reward window that will slide over time
and what we want to observe is a mean
of the last 100 rewards increasing with time.
So let's initialize it
with self dot reward underscore window.
And so since this is going to be a sliding window
of the evolving mean of the last 100 rewards,
well, we are going to initialize it as an empty list
and then we will append the mean of the rewards over time.
All right.
Then more exciting, let's create our neural network.
So, we're gonna call it self dot model
because basically that's the heart of the model,
so I'm calling it model.
And this model is going to be nothing else
than an object of the network class.
And to create such an object,
we take our class network then parenthesis,
and here, we just input the arguments of the network class.
But we put these arguments
in the arguments of the init function
and therefore, we just need to copy them right here
and just paste them in the network class.
And there we go, with this line of code
we create one neural network for our Deep Q-Learning model.
Perfect. Then, let's create a memory.
So again, we're going to create a new variable
that we call self dot memory.
And again, this is going to be an object
of the replay memory class.
So let's just take the name of our class,
let's copy it, let's paste that here.
And in some parenthesis, we need to input the capacity
because the capacity is an argument of the init function
and that's the only argument we need to input here.
So, what capacity are we going to choose?
Remember that corresponds to the number of transitions
the number of events, last state, new state,
last section and less reward.
And so, as mentioned in one of the previous tutorials,
we're gonna take 100,000.
100,000 transitions into memory
and then we will sample from this memory
to get a smaller number of random transitions.
And that's on which the model will learn.
Okay, so now we have our memory.
Perfect. Now let's get our optimizer.
So again, self,
we create a new variable that we call optimizer.
So optimizer is another variable of our future DQN object.
So, self dot optimizer.
And now if we go back up,
you can see that we imported torch dot optim
which is a module of torch that contains all the tools
to perform stochastic gradient descent.
So of course, it contains some optimizers
and we gave it the shortcut optim.
And therefore, here's what we're gonna do
is take the model optim, which is torch dot optim,
and from this module,
we're gonna take one of the optimizers.
So as you can see, they're all listed here.
Many of them are excellent.
For example, RMSprop is an excellent optimizer
that is for example,
highly recommended for a rec renewal networks
or unsupervised deep learning
but the other one that is excellent
and that we will choose is the add-on optimizer,
that's the one.
You'll see that with this one,
we'll get a good self-driving car.
But again, you are totally welcome to try other ones,
you can try the RMSprop
but for our model, we will choose add-on.
So, I'm pressing enter.
And in fact, you notice there is the capital A here,
that's because we are creating an object
of the add-on class, this is a class,
but the object will be an add-on optimizer itself.
But since this is a class, we need to input some arguments,
the arguments of the add-on class.
And the arguments are all the parameters
that can customize your add-on optimizer.
So for example, that's typically the learning rate,
the decay or some other parameters.
And besides taking all the parameters of our model,
we will specify a learning rate.
So, speaking of the parameters of our model,
we can get them with self dot model.
So, that's the model we created here,
self dot model from our network class.
So, self dot model.
And then, to access the parameters of the model,
we add another dot and then parameters
with some parenthesis, very simply.
So, that's just to connect the add-on optimizer
to our neural network, the one that we created here.
Okay, and then as we just mentioned,
we're gonna add a learning rate,
and the argument for this is L R.
And we will set it equal to a value
such that the learning doesn't happen too fast.
If we get a learning rate too large,
then the AI won't learn properly.
We want to give our AI some time to explore,
learn from its mistakes.
You know, when we punish it, when it's making some mistakes,
like going onto some sent or getting too close to a wall.
Well, we want to give the AI some time to learn.
We want to wait at the neural network
to update correctly.
And so, a good value for the learning rate
I ended up with, have to train several of them,
is 0.001.
All right, and that's all we need to create an optimizer.
So basically, we are creating an object of the add-on class.
Great, and then, the last three variables we need
are the variables composing, are transition events.
So that's the last state, the last action,
and the last reward.
And so, that's basically, what we'll create now
and we will just need to initialize them.
So, let's start with the last state.
The last state,
we're gonna call it self dot last underscore state.
And then, how we going to initialize it.
Well, remember, the last state
is a vector of five dimensions,
a vector that is encoding one state of the environment.
And as a reminder, these five dimensions
are the three signals of the three tensors
left straight and right
and orientation and minus orientation.
So, this is a vector in the intuitive sense
but for PyTorch, it needs to be more than a vector,
it actually needs to be a torch tensor.
But not only it needs to be a torch tensor
but also it needs to have one more dimension
that I like to call a fake dimension
that corresponds to the batch.
And that's because the last state
will be the input of the neural network.
But when working with neural networks in general,
whether it is with TensorFlow, Keras or PyTorch,
well, the input vectors cannot be a simple vector by itself,
it has to be in a batch.
The network can only accept batch of input observations.
And therefore, not only we will create a tensor
for our input state vectors
but also we will create this fake dimension
corresponding to the batch.
So let's do this
and let's start by initializing a torch tensor.
So to do this, there is nothing more simple.
We take our torch library,
then dot and then we're gonna use the tensor class
because as you might have guessed,
this will create an object of the tensor class
that is a tensor object.
And in this tensor class, we need to input one argument
which will specify the size of your tensor.
You can picture a tensor like an array,
having one single type.
But basically what this will represent now
is of course, this input state,
which you can see as a vector.
And so, to specify the number of elements
this tensor must have,
well, we need to use of course the input size
because the input size is exactly the number
of dimensions of our input state vectors.
Now, I should say tensors.
And so, what we simply need to input in our tensor class
to create tensor object, well, that's input size.
And later on, input size will be equal to five.
All right, so that's good.
That's the first thing done.
We just initialized a tensor as it should be.
But then remember, we need to do another thing,
we need to create that fake dimension
because this is what the network will expect for its inputs.
And to create this one fake dimension,
which by the way, has to be the first dimension,
you know, the fake dimension corresponding to the batch
will be the first dimension of this last state variable.
Well, to do this, we simply need to add dot then unsqueeze.
And then, and some parenthesis,
we need to input the index of this fake dimension.
And as I just said, this fake dimension
has to be the first dimension of the last state.
And since indexes in Python start at zero,
we need to input zero, so that this new fake dimension
is becoming the first dimension.
So, we have a first dimension corresponding to the batch
and then the dimension corresponding to the tensor
which will contain the five elements of your input states,
these three signals, orientation and minus orientation.
And there we go,
we initialized our input states properly.
Perfect.
And then two variables to go and that's gonna be much easier
because the next variable is the last action.
That's a new variable we're creating for our object,
last action.
And remember, in the first tutorial of this section,
I told you that the actions
are gonna be either zero, one or two.
And then, using the action to rotation vector,
we will convert these indexes of these actions
into the angles of the rotation
which I remind are zero, 20 or minus 20.
We can actually, refresh our memory with that.
Well, it is exactly here, action to rotation.
If the action is zero,
well, this will correspond to the first index here, so zero.
If the action is one,
this will respond to the index one of this vector,
so 20 degrees.
And if the action is two, we will get minus 20 degrees.
That's gonna be the rotation angle of our car
when we play the action.
All right?
And therefore, since the action is going to be either zero,
one or two, well the action is therefore a simple number.
And so very simply, we can initialize it to zero.
We don't need to create any tensor here or anything else,
we just need to initialize it with zero.
And finally, well, that's the last reward.
So, self dot last reward.
There we go.
And again, the reward is a float number
which I remind is between minus one and plus one.
So, that's the number again.
And as for the action, we will initialize it to zero.
And there we go.
Congratulations, our init function is ready.
So, now we are ready to move on to the exciting stuff.
And actually, the most important thing for our AI
that's deciding which action to play at each time,
at each time T.
And that's exactly what we're gonna do in the next tutorial
by creating the select action method.
So, let's do this in the next tutorial.
And until then, enjoy AI.