-: Hello and welcome to this tutorial.
In the previous tutorial, we initialized the map.
And now, time for the exciting stuff.
We create the car.
And we do that with a class of course.
You will see that the class is very practical
to create some things that have a lot of properties,
because as you can see,
not only I'm defining some variables for my car,
but also some functions,
which of course is the function that will make the car
move to the left, to the right, or going straight.
So we have a couple of variables
that are important to describe the environment.
We have, for example, the angle,
which is the angle between the X-axis
and the axis of the direction of the car.
Then we have the rotation, which is its last rotation,
which remembers either zero degree,
20 degrees, or minus 20 degrees.
Then we have the velocity,
the X-coordinate of the velocity vector
and the Y-coordinate of the velocity vector,
and then the vector
of coordinates velocity X and velocity Y.
Then we have the sensors and the signals,
and that's very important.
The car that we're making will have three sensors,
sensor one, sensor two, and sensor three.
Sensor one will be detecting
if there is any sand in front of the car.
Then sensor two is the sensor that will detect
if there is any sand at the left of the car.
And sensor three is the sensor that will detect
if there is any sand at the right of the car.
And then from these three sensors, we get the signals,
that is the signals received by each of the sensors.
So signal one is the signal received by sensor one.
Signal two is the signal received by sensor two,
and signal three is the signal received by sensor three.
And so how does it work?
Signal one is the density of sand around sensor one,
signal two is the density of sand around sensor two,
and signal three is the density of sand around sensor three.
And how do we compute this density of sand?
Well, that's very simple.
We take some big squares around each of the sensors.
These are actually squares of 200 by 200.
And for each of the squares,
we divide the number of ones in the square
by the total number of cells in the square.
That is, 20 times 20 equals 400.
And that gives us the density of sand
because the ones correspond to the sand.
We do this for each sensor,
and that gives us the density of sand
around each sensor, that is, the signals.
All right, so now we have everything to detect the sand
and then we have the move function.
And of course, the move function is what will allow the car
to go to the left, going straight, or going to the right.
So let's go through it quickly.
We have here the update of the position of the car
with its last position, which is self.pos here,
and the velocity vector.
So thanks to the velocity vector,
the position will be updated
in the direction of the velocity vector.
Then we get the rotation,
which we will get further down in the code right here.
Rotation equals action-to-rotation action.
Here we will select the action
and then getting the rotation.
And so, this self-rotation equals rotation here
is this rotation that we get
to know how we need to rotate the car
that is going to the left or to the right.
Then we update the angle,
which I remind is the angle between the X-axis
and the axis of the direction of the car.
And then once the car has moved,
then we have to update the sensors and the signal.
Because of course, when the car has just rotated,
well, the sensors have rotated as well,
and therefore we need to rotate them
by using the rotate function
and to which we add the new position.
And why do we have this vector of 30,0?
Well, that's simply because 30 is the distance
between the car and the sensor.
You know it is the distance between the car
and what the car detects.
And then once the sensors are updated,
well, then it's time to update the signals.
And so here we do exactly what I explained
to compute the signals.
We get the X-coordinate of our sensor.
Then we take all the cells from minus 10 to plus 10.
Then we do the same for the Y-coordinate,
taking all the cells from minus 10 to plus 10.
So therefore, we get the square of 20 by 20 pixels
surrounding the sensor.
And inside this square, we sum all the ones.
So basically, we sum all the cells,
because the cells contain either zero or one.
And since
in a 20-by-20 square, there is 20 times 20 equals 400 cells,
well, we divided by 400
to get the density of ones inside the square.
And that's how we get the signal
of the density of sand around the sensor.
And we do the same for the second sensor
and the third sensor
to get the second signal and the third signal.
Okay, so that's to detect the sand.
And then the three lines of code here are very important.
It's another bad reward that we want to give to our car
when it is reaching one of the edges of the map.
We don't want the car to rush into some walls
and therefore we want to penalize it,
to punish it when it's getting too close to a wall.
And therefore, that's what we do here.
If the first sensor is larger than longer minus 10,
that is, larger than here,
because longer is this distance here.
So longer minus 10 is right here.
So if sensor one X larger than longer minus 10
concerns all the points that are here,
that is, if the car is getting closer
to the right edge of the map.
Or if cell sensor one X lower than 10, that's right here,
if the car is getting closer to the left edge of the map.
Or if sensor y is larger than larger minus 10,
that's the upper edge of the map.
And or if self-sensor y is lower than 10,
that is, the lower edge of the map.
And so if the sensor one is reaching
any of these four edges,
well, we will put the signal of the sensor,
signal one is the signal of sensor one,
we will set it to be one.
And what does that mean?
That means full sand, like the full density of sand.
It's like the worst sand you could get.
There is so much sand that it's going to stop your car.
So signal will be one,
and therefore the car will get a terribly bad reward.
All right, and then we do the same for signal two
and signal three from sensor two and sensor three.
All right. And then we create the Game class.
So that's basically the class to create the game
because so far we have only created the car.
And now of course, we have to create the map.
We have to create the game itself.
So we will not be playing the game.
It's our AI that will be playing the game.
And the game is actually to avoid the obstacles
and to go from the airport to downtown and vice versa.
So in this Game class,
we need to create some objects, like the car.
Then we need to define the update function.
That is the most important
and actually we will focus on that right now,
because that's in this update function
that we will select the action that the car has to do
at each time to accomplish its goal.
And this action is exactly the output of our neural network,
the neural network that will be at the heart
of our artificial intelligence.
And so this action is returned by the brain of the car,
which I remind is the object of our DQN class
that we'll be making in our AI file.
And this object has a method that is called update,
and it takes as input the last reward and the last signal.
So the last reward is of course the last reward obtained
by the car.
And the last signal is of course the last signal
of the three sensors.
Signal one from sensor one.
Signal two from sensor two.
Signal three from sensor three.
But then I'm adding two other inputs,
which is the orientation of the car
with respect to the goal.
So for example, if the car is heading towards the goal,
then the orientation will be equal to zero.
If it goes slightly to the right,
then the orientation will be close to 45 degrees.
And if it goes slightly to the left,
the orientation will be close to minus 45 degrees.
So that's the fourth input of our input state.
And then there's a last input, which is minus orientation.
So usually, the inputs of a neural network are independent.
There is no multicollinearity.
But it doesn't really matter if we add this
because the neural network will just fix that
with the weights.
But still, I noticed that by adding this minus orientation,
well, that allows the car,
the training of the car, to stabilize the exploration.
We're doing this so that the AI doesn't always explore
in the same direction.
By adding this minus orientation,
we make sure that it explores in both directions,
right or left.
And so this,
the three signals plus the orientation and minus orientation
are the five inputs of our encoded vector,
which will go into the network.
That's our input vector that will go into the network.
And after it goes into the network,
well, the network returns the output,
which is the action to play at each time.
And the output is returned by this update function
that contents the network itself
and the output of the network.
And therefore, that's why we have to input the last signal,
that is, the input state,
and also the last reward
because the action to play also depends on the last reward.
All right. And then we update the mean score of the rewards.
We update the rotation.
We use the move function to rotate the car
according to the action that was selected.
We update the distance of the car to the goal.
And we update the positions of the sensors.
Ball one, ball two, and ball three correspond to the balls
that will represent the sensors on the map.
You will see that very quickly.
And then here, that part is very important
because that's where we penalize the car
if it goes onto some sand.
Because as you can see, this means
if the car is onto some sand, well, it will be slowed down.
So that's where we reduce its velocity.
Its velocity is usually six, as you can see here.
And if it goes onto some sand, it will be one.
So it will be slowed down to one.
You'll see how the car will be slowed down
once it goes onto some sand.
So it is slowed down,
and besides, it gets a bad reward.
It gets a minus one reward.
And that's actually the worst reward you could get.
The best reward is one, the worst reward is minus one,
and the reward is between minus one and plus one.
And then otherwise, if the car isn't onto some sand,
well, it keeps its usual speed, speed of six.
And then we add something else.
If it's getting closer to the goal,
then it will get a slightly positive reward.
And if it's getting further away from the goal,
well, it gets a slightly negative reward, minus 0.2.
And then last conditions that are related to the reward.
Well, that's if the car is getting too close
to one of the edges, as we spoke of earlier.
Remember when we talked about full sand?
Well, if the car is getting too close
to the left edge of the map, it gets minus one reward.
If it gets too close to the right edge of the map,
it gets reward minus one.
And if it gets too close to the bottom edge of the map,
it gets reward minus one.
And if it gets too close to the upper left of the map,
it gets reward minus one.
So that's a terrible punishment.
And so you will see how it will learn fast
not to rush into some walls.
All right, and then this is to update the goal
when the goal is reached.
So when the car reaches the airport,
which is the first goal,
that is the upper-left corner of the map,
well, the goal changes
to the bottom-right corner of the map, which is downtown.
And that's exactly what we do here.
We update the X-coordinate of the goal
and the Y-coordinate of the goal,
and then we update the distance from the car to the goal.
All right. And then that's less important.
That's just a class that will add the painting tools
for us to be able to paint some roads
or some obstacles on the map.
That's more related to SkyV.
You can have a look if you want.
I'll provide the commented version of this code
and I'll provide some reference
if you want to go deeper on how to do that with SkyV.
But we're getting further from artificial intelligence,
so I'm not gonna go into the details of it.
And that's the same for the last code section here
with the Car app class.
That is just to add the API buttons, clear, save, and load.
So that's what we do here, clear canvas, save.
And that's actually very important.
That's for us to be able to save the AI,
to save the brain so that you can reuse it later
by taking the load function,
which is another tool we add on the map,
to load the brain of the car,
that is, to load the memory of the car,
how to navigate in the map.
And then finally, we have the last of the last code section,
which runs the whole thing,
that is, which runs the map and the AI itself.
And actually, that's what we're gonna do right now.
Let's have a look at everything we make in this code.
So right now, the AI is not implemented
so the car will have a very random movement.
It will actually look like an insect.
But don't worry. We will fix that.
Not only we will train it to move like a real car
and train it to navigate,
following some roads and avoiding some obstacles.
So let's do this.
I'm going to select everything and execute.
And here's the map and here's the car.
All right, so that little thing here that you see
that looks like an insect is our car.
So as I told you, the actions are totally random.
So at each time, the car selects randomly an action
whether to go straight to the left or to the right.
So that's why it is making some nonsense movement
and that's why it's looking like an insect.
So we will fix that, of course.
And of course, since the AI is deactivated,
well, it is not going to the goal,
which is the airport here,
or to downtown at the bottom right of the map.
And we will fix all this by making the AI.
So we will implement the AI into this car or this insect.
So you can see the three balls here,
the yellow one, the red one, and the white one,
that's our three sensors.
So that's what will detect if there is some sand around it.
And speaking of sand, well, let's draw some.
So to do so, I just need to do a click left here.
And drawing some sand by still clicking left.
So right now, I'm adding some sand.
We can add some more.
So each time I'm adding sand, as you can see,
that's putting ones in the sand array.
That's the sand array.
That's the zero, zero coordinates of the origin.
And here there are a lot of ones.
And as you can see, well, that's good to see the car
just went onto the sand and was slowed down.
So as you can see right now,
it is really slowed down because it's going onto the sand
and right now it's trying to escape.
And so what we'll do is we will draw some roads
and we will draw some roads from the airport to downtown
and we will train the car to stay in the road
and to avoid the obstacles.
All right, and as you can see, there is the clear button
to clear the sand.
There is the save button to save the brain of the car.
And actually, there is the score curve that we spoke of.
So that saves the AI,
that saves your model actually, the brain of your car.
And then you can, when you leave your code
or turn off your computer
and you want to go back to it again,
you can use the load button to load your model,
that is, to load the brain,
and that will get the trained AI of your car.
All right, so now I can't wait to start making the AI.
This will be a lot of fun.
We will make our neural network
and we will punish the car
as soon as it doesn't do what we want.
So let's do that from the next tutorial.
And until then, enjoy AI.