-: Hello and welcome back
to the course on artificial intelligence.
And finally, we are onto the fun stuff.
We're onto deep key learning.
All right, so let's have a look.
Previously we spoke about key learning and what it's all
about and we learned about the agent, the environment
and how the agent will look at the state he or she is in
take an action, get a reward, enter into a new state.
And based on that feedback loop
they will continue taking actions
and they'll learn from that.
Understand what are the better actions to take.
And so we looked at this basic example of a maze.
We understood that as the agent explorers environment
and understands what the values of the states are
then we moved on from dealing with the values of the states
to dealing with the values of the actions or the Q values.
And then based on that, we understood how plans
in a non stochastic environments work
and how policies work in stochastic environments,
and this is an example of policy.
So that's a quick recap
of everything we discussed in the basic Q learning.
And now let's have a look
at how this can be taken to the next level
through deep learning, through adding deep learning.
Okay, so this is our environment,
and what we're going to do now is we are going to add,
instead of just doing basic calculations
in this matrix that we have, which is pretty, pretty simple
what we're going to do is we're going to add two axes.
We're going to add an x and y axis, or we'll call them X one
and X two, just to make things even more general.
And here we've got to, we'll number the columns 1, 2, 3, 4
here we'll number the rows 1, 2, 3.
And so now every single state can be described
by a pair of two values, X one and X two.
So any one of these squares
in which the agent can possibly be in,
can be described by X one, X two.
So for instance, right now he's in the square
with X one equal to one and x two equal to two.
And therefore that's in the same
that same way we can skip any square
meaning we can describe any state.
And of course this is a very simplified version
of an environment of describing states, but nevertheless
it works in this case.
And that means that now we can feed
these states into a neural network.
And by the way, here, I would just like to mention
that at the end of the course we've got annexes
we've got annex number one and annex number two.
In order to proceed successfully with this section
it's highly advisable that you check out annex number one,
which is on artificial neural networks
so you understand how they work
so that we can, we don't have to delve into that here
and we can just use the benefits
of the knowledge of how artificial neural networks work.
And so we feed in this information
on the state into a neural network
and then it will process this information.
So X one and X two
depending on the structure of the neural network
it might have multiple hidden layers and so on.
So that's something that you'll figure out
in the practical tutorials.
-: But at the end, we will structure
in such a way that it spits out four values.
And these four values are actually going to be RQ values.
So the values which dictate which action we need to take.
And further down in this tutorial, we'll see exactly
how these Q values are used to decide which action is taken.
But the main point here is that we no longer look
at just this maze from a Q learning perspective.
We're now taking the states
of the maze and we're feeding them into
a deep neural network in order to get these Q values.
And, and at the end of the day
we are still going to come up with an action.
We're still going to understand how
what action we need to take
and we'll discuss all this in more detail.
But the question right now is why.
Why are we doing all of this?
Why are we making things so
much more complicated when that initial approach
of Q learning was working already?
Well, the reason for that is the Q learning was working
in this very simplistic environment
and we're continuing to deal
for now with this very simplistic environment
in order to better understand the concepts.
But at the same time, that simple Q learning
will no longer work in more complex environments.
And we're talking about, for instance
the self-driving cars which you'll be creating,
or playing doom when the other artificial intelligence
is playing doom or other Atari games
like breakout or even self-driving cars
and more advanced reinforcement learning things such
as like robots walking around and performing actions.
In all those cases, basic Q learning is insufficient,
is not strong, is not powerful enough to
be able to master those challenges.
And just like pro we've seen in the deep learning course
if you've been in our deep course,
or if you've done the annex sections
annex number one and X two, you will already know
that deep learning is by far superior to any type
of machine learning, let alone based simple Q learning.
And that's why we're leveraging the power
of deep learning here.
So we're feeding in the information
about the environment as a vector of values.
So in this case just to values into a deep neural network.
And then we're using that to perform the actions
that we want to decide
which actions the agent's going to take.
So that's kind of
like a high level overview of why we're doing this.
And now let's have a look at
in a bit more detail at what happens
to the concepts of key learning when we
make this transformation from or transition from
simple key learning into deep key learning.
So as you saw in the previous intuition tutorials
we had a slide like this
which is the foundation of temporal difference learning.
This is the formula for temporal difference.
And basically, so let's go through this.
So basically we had an agent who was
in this state over here, which is
indicated by the blue arrow.
And we were understanding how temporal difference works
for this Q value of for instance, going up.
And so what we saw here was before
this is in the simple key learning
not the deep key learning this in the simple key learning.
What we saw was
before the agent had a certain Q value that he had learned
about this action of going up.
And so then he decides to take this action to go up.
And right after he takes this action
he gets a reward for taking this action in this state.
And that is that reward.
Plus, now he can evaluate the value
of the current state he's in, which is the maximum
of all of the new Q values
of all of the q valves of the new actions
He can take a prime in the new state S prime.
and we multiplied by the DK factor of gamma.
So that is essentially the Q
the new Q value or kind of like the
the empirical Q value that he has just received
for taking that action.
And ideally these two two should be the same.
So the Q value that he had in his memory
about this action in this state should equate
to the actual reward plus the gamma
times the value of the state that he ended up in.
And therefore that's how we calculated the
temporal difference.
We take what he got after minus what he got, what he had in
mind, what he was expecting, you'd subtract one
from the other and that's your temporal difference.
And then you use your learning rate alpha to
adjust your Q value your new Q value
by the temporal difference, but with a coefficient of alpha.
So that is the essence of the simple key learning.
Now let's have a look at how it changes
in deep key learning.
And so we're still gonna work with the slide,
but we are going to just see exactly what's happening.
So in deep Q learning, the neural network will predict
for valves as we saw in the previous slide,
and as we'll see further down in this tutorial,
the neural network will predict four values,
or it might predict more values
if there's more possible actions in a given state.
But in this case we know
that there's only four actions: up, right, left, down.
And so the neural network will predict four of these values.
So there will be no in a deep Q learning situation.
It's important to understand there's no before or after,
and this is how we'll get to know this a bit better.
So the neural network will predict four of these values,
and it will compare not to what will happen after,
but the neural network will compare to this exact value,
but it was this value which was calculated
in the previous step.
So in the previous time when the agent was
in this exact square, so let's say, I don't know,
some time ago the agent was again,
was in this exact square as well,
and it calculated this value previously.
So in the previous time, long time ago
the agent calculated this value
then the agents stored this value for the future
and now the future has come.
So now he's in the square again
and now he's got these Q values, which is predicted
and one of them is for going up.
So now what he's gonna do
he is gonna compare the predicted value of Q to this value
which he had recorded from the previous step.
And we'll understand exactly why
this is important right now.
So just important to understand here is there's no before
and offshore in this specific square
this specific time, we're taking the Q value
that he's predicted using the neural network this time.
And we're comparing it to this value which he had
from the previous time, from the previous time he was
in this square assessing all of the situation.
And you know, like the previous time
he actually performed this action.
So there we go.
Now let's have a look at how this all works
out in the neural network and why is it like that?
I know it sounds a bit complicated right now,
but we'll break it down into simple terms, right?
Just in a second.
So this on neural network, we're feeding in the states
of the environment into the neural network is going
through the hidden layers, then it's coming out
with these outputs, q1, q2, q3, q4 in that specific state.
These are the Q values
that's the neural network is predicting
for the possible actions.
Those are the Q values.
So then we're comparing to target,
and these targets is exactly
so if we go back here, this is the target.
So this is the value that was predicted.
And then, but also we know that we have a target
from the last time we were in the square.
We have a target for this same action,
which is up for instance.
So here we've got a target and we're going to compare
So we're comparing Q1 versus that target.
We're comparing Q2 versus that target
the target that we had from previously,
Q3 versus the target, Q4 versus the target.
And so this is the part where the neural network
or the agent is now learning
through deep learning how to better go through space.
And the key point here is
that we are still applying key learning
but the concepts in simple key learning
you learn through temporal differences,
which are pretty straightforward,
which we've already discussed and we know quite well by now.
But at the same time in deep learning
how do neural networks learn?
Well, neural networks learn through adjusting their weight.
So we have to adapt the concepts of simple key learning
to the way neural networks actually work.
And that is through updating their weight.
And so this is what we're trying to figure out here
how do we adapt that concept
of temporal difference to a neural network
so that we can leverage the full power of neural networks?
And so far we've gotten this.
So we enter our environment state here as a vector goes
through a neural network, we get predictions of Q values
and then from the previous time the agent was in that state
we have these Q target, Q target 1, 2, 3 and 4
for each of these respective actions.
And so now we are up to, okay
let's compare each one with each one.
And from here it becomes pretty straightforward
if you are up to speed with neural networks.
Once again, that's all in a annex number one,
we're going to calculate a loss which is L here,
and we going to be Q target this one minus Q minus this one,
we're going to square that.
So the square difference
of each one of these, and we're gonna sum them.
So we're gonna take the sum of the squared differences
of these Q values in their targets
and we're gonna sum them up and that's gonna be our loss.
And so ideally, just as we had
in the temporal difference learning,
so if we go back for a second,
remember we said ideally we want this to be equal to this
so we want the temporal difference to be zero.
So that's, that means basically the agent is
predicting correctly what's, you know
the Q values that the agent is predicting are exactly,
or that he has a memory, are exactly descriptive
of the environment and therefore the agent can navigate
the environment pretty well, right?
There's no surprises, there's no,
if as long as the temporal difference is highly positive
or highly negative, then then we've got some surprises.
But if the temporal difference is zero
then he knows the environment so well
that he can predict what's going on and he can,
and therefore his policy is going to be very good
and he's going to be able to navigate it.
So here, same thing.
So we want this loss to be as close
to zero as possible, as small as possible.
And that's why now we're going to
this is the part where we are going to
leverage the real true power of neural network.
So we're gonna take this loss
and we're going to use back propagation
or stochastic gradient descent to take this loss and pass it
through the network, pass it back or back propagate it
through network and through stochastic gradient descent
update the weights of these synapses in the network
so that next time we go through this network
the weights already a bit better descriptive
of the environment.
And that's exactly how it works.
So here you have, if we go back, this is calculated,
loss is calculated and it gets product proof propagated
for the network, the weights are updated
then the next time we get here, this happens again
and we get here, this happens again and so on.
And so, and it keeps happening and that's how
this agent learns or basically now it's the neural network
which is the brain of the agent is learning,
is becoming more
and more descriptive of the environment and therefore
the agent is able to navigate the environment better.
When we say descriptive environment basically means
that when we put in the states
of the environment that the agent is in
we are more likely to get closer
and closer to the actual Q values.
And that happens because the Q values that we want
and find the right action.
And that happens because these Q targets
are actually empirically derived.
So how does he find these Q targets?
That's actually this.
So he actually observes,
"okay, so once I do take this step,
what's the reward I get?"
And then " what's the value this of this state?"
So same thing as we saw previously
in key learning in the simple key learning intuition.
So he learns this through trial
and error and then he constructs his network
or updates the way his network
in such a way that the predicted Q values are close
and closer approximating the target Q values.
So very similar to the concept we discussed here
in the simple temporal difference learning
of the simple Q learning algorithm.
So there we go.
That's how the agent learns.
So we're up to here.
That's the learning part.