Instructor: Hello, and welcome back to the course on AI.
In the previous part,
we talked about the key learning intuition.
We started there.
And in fact, we actually got all the way to this part
and where we talked about learning.
And now we're going to move on to the actual acting part.
So there's two parts, two distinct parts
that we have to remember.
So, that's the learning part,
but now he's done all of this, that's beautiful.
Now he actually has to take an action.
He has to decide what is he gonna do?
Is he gonna do action one, two, three, or four.
And so, how does he do that?
Well, the way he does it is now given those same Q-values,
so the Q-values don't change.
After we have these Q-values, we've compared them,
we've calculated the loss, we've propagated the error,
we've updated the weights,
but the Q-values don't change in that whole process.
So after we've got the Q-values, they're fixed.
We know what they are.
So this happens, the network's updated.
And now using those same Q-values that we had
what we're going to do is we're going to pass them
through a Soft Max Function.
And again, Soft Max is described in, I think in annex two.
And we'll talk a bit more
about Soft Max further down in,
or we'll talk about this action selection policy
further down in the rest of this section.
So just in the future tutorials,
but for now, we're just gonna just say we we're passing it
through the Soft Max Function.
And basically what it does is it allows,
it helps select the best one.
It selects the best action possible.
And there's a small caveat to that.
It's not just the best one possible.
We'll talk about that
in the action selection policy tutorial.
But for now, let's just say
it selects the best action from here.
It says, okay, so Q1, you know, the likelihood
basically we know the Q-Values,
so it predicted the Q-Value.
So it can look at them and say,
okay, so the highest Q-Value out of these,
just as we did in the simple Q-learning algorithm,
it'll just look at all these four,
and say the highest Q-value is this one,
and I'm gonna select that action.
We're gonna take those.
And that's pretty much it.
That's how it chooses which action to take,
takes the action
and then all of this process happens again
for the next state.
The ad agent ends up in,
in our case, in the next square of the maze,
but generally speaking into the next state.
So there we go.
That's how we feed
in a reinforcement learning problem into a neural network
through a vector describing the state that we're in.
And once we feed it in,
there's two parts of the process that happen.
Part one is the learning.
So, remember that part where we compare each of the Q-values
with the targets and then we back propagate the loss
through the network to update the weights
so that our network is learning
as we go through this maze
or through this environment.
And also the second part is, of course, we have to act,
we have to select an action,
and that is where we pass the Q-values
through the Soft Max Function,
and or basically an action selection policy,
which we'll talk about further down.
And then we simply select the action that we want to take
and we perform that action.
And then this whole process starts again.
And then maybe the agent gets to the end,
maybe the agent doesn't pass the game,
in any case, the game ends.
And then once again, the whole process repeats.
The agent plays the whole game again
and then that stops.
So basically, that's another epoch every time the agent,
you know, every time the game ends,
whether favor beyond favor,
that's the end of an epoch.
And then he starts again,
and then he starts again,
and then he starts again, and so on.
So, that happens and this process happens
for every single time
the agent is in a new in a new state.
So the state is encoded here.
So that's important.
So not just for every single game that he plays,
but for every single state.
So he's in a state, it goes through his process,
it updates, and so on,
and happens every single time.
And so the learning happens,
and then the acting happens as well.
So that is Deep Q-Learning in,
oh the intuition behind Deep Q-Learning.
We've got lots more to cover off.
And then of course we've got practical.
And in the meantime,
if you'd like to get some additional information
on Deep Q-Learning, we've got a recommended reading.
So, we've already spoken about
Arthur Giuliani's series of blog posts.
If you look at simple enforcement learning
of potential flow part four,
you will find the part that's relevant
to what we discussed today.
Note that here, he talks about convolutions.
We are not covering revolutions in this section.
We're gonna be talking about them
in the next section of the course.
So, the difference here is that,
so just kind of skip the convolutions part for now
and we'll talk about them in the next part of the course.
But the difference is in convolutions you're like looking,
your agent is looking at the image
and therefore he has to process an image.
So, an additional complication.
For now, we're slowly gradually building up to that.
For now, we are encoding our environment through.
So if you look here, we're encoding our environment
or maybe like look at this one,
probably encoding our environment as
a or encoding a state the agent is in as a vector.
So in our case, it was a very simple of two values.
But sometimes people even in that simple maze,
sometimes, or as you'll see from this blog post,
sometimes people prefer the one hot
encoded version of that state.
So basically, where every single box of the maze has a,
so you have like a vector,
for in our case it would be 12 values,
three by four.
So, it's like either one or a zero,
depending on which elements,
which box you're in in the environment.
So whichever way you decide to encode your environment
and the state of your environment,
that's how we're encoding it.
So, it's basically a vector.
The key here is that it's not a convolution,
so it's not like an image
and there's no convolution involved.
So this part will come later.
For us, it starts over here.
And that just simplifies the process
for us to gradually understand better.
And of course, don't forget that this blog post
is really an intensive flow
and we are using Pie Torch in our tutorials.
So, hopefully you enjoy this quick intro
into a Deep Convolutional, not convolutional,
yet Deep Q-Learning.
And on that note, I look forward to seeing you next time.
And until then, enjoy artificial intelligence.