In this lecture we are going to extend our knowledge of basic learning to deep learning that is Q learning
that uses neural networks to understand the motivation behind this.
Consider what we have been doing so far.
Q. learning involves finding the optimal Q of S and a and the corresponding optimal policy.
As you recall just like with supervised learning targets we can think of the states as categories and
the actions as categories in a computer.
We would represent them with integers starting from zero.
For this reason we call Q of DNA a Q table it's a table as in something that looks like a data frame
or a matrix.
Q zero zero means the action value for state zero and action zero.
This is nothing but a two dimensional array methods which use Q tables are called tabular methods for
obvious reasons.
In this section we haven't spent too much time explaining in detail how to implement tabular methods
because my goal was to get you to deep Q learning as quickly as possible.
However it's important to realize that what we have been discussing so far was in the form of tabular
methods so you can understand why we want to move on from them
consider the scenario where your state space or your action space is continuous or infinite.
It should be clear that storing a Q table is no longer a viable option.
We cannot have an infinitely sized table.
One option which would be relatively simple to implement is to use the building approach to force your
state space or action space to be discreet.
For example suppose you're playing the inverted pendulum game in one of your states is the angle of
the pendulum the angle of course must be between zero and 2 pi.
Thus we can do something like break this up into equal sized bins of size pi over four so any angle
between a zero and pi over four would be considered category 0 pi over for it's a pi over 2 would be
considered Category 1 and so on.
This is a fine approach but it can only take you so far
instead.
If we want to do something more flexible we have to make use of machine learning.
We call these approximation methods because we are using machine learning models as function as for
deep Q learning.
We will consider the case where the state is continuous but the action is discrete because the state
is continuous.
We can't use a cue table.
Instead let's suppose our state is represented by a continuous vector s and let's say for simplicity's
sake that we have two actions a zero and a one in this case we can represent U of S a zero as the dot
product between W 0 and s plus B zero we can similarly represent Q of S and a 1 as w 1 transpose s plus
b 1.
In other words we can approximate the Q value for both actions as linear regressions.
Of course there is no need to represent the weights separately.
Instead we can combine all the weights into a weight matrix and combine all the biased terms into one
bias vector.
If the state is a continuous vector of size D and we have k different actions then the weight matrix
will be of size D by K and the bias vector will be of size k.
Notice how I'm abusing notation here a little bit by mixing it num pi notation into our math the coal
in it means select all values in this dimension.
So this is the CU value given the state s overall actions
one example of this is the inverted pendulum task in this task.
Your state consists of four components the position of the car the velocity of the car the angle of
the pendulum and the angular velocity of the pendulum.
We can write this using the notation x x Di theta theta.
The action in this environment is the amount of force to apply to the car.
Although this could be continuous since force is a continuous variable environments like carpool in
open a gym usually descriptors this force.
So you can only apply a force of either plus 1 or minus 1 in this case for the action value.
The weight matrix would be a size 4 by 2 and the bias vector would be a size 2.
If we're predicting the state value V of S then the output is a scalar.
In this case the weight would be a vector of size 4 and the bias would be a scalar
previously when we discussed tabular temporal difference loading we would update V of S and queue of
assigned a directly because those were just entries in a table now.
Since Q and V are not tables we can't do that.
So what do we do instead.
Well it's clear that what has to be updated are the weight matrix and biased vector
let's start with the prediction problem finding V of s here is where treating the problem as a supervised
learning problem becomes important.
What is the target.
What is the prediction and what are we trying to update.
Let's list these out the target is the return or in the case of Q learning.
It's the estimation of the return r plus gamma times V of us Prime.
One exception to this is if s prime is a terminal state.
We know that a terminal state represents the end of an episode.
So there are no further states and hence no further rewards.
Therefore if s prime is terminal the target is simply the reward.
R next what is the prediction.
The prediction is v of S and what do we want to update.
We want to update the linear regression parameters W and V
since this is a regression.
We know that we should use the squared error and do gradient descent to update the parameters.
Let's see how this will work.
First we set up our loss J equals R Plus gamma times V of as prime minus V of us all squared.
Next we find the gradient of J.
With respect to the parameters
one important thing to realize here is that even though both a V of s and view of us Prime depend on
the model parameters we only differentiate V of s with respect to W and b we pretend that V of US prime
is constant.
Since it's part of the target and we do not differentiate with respect to the target
so let's see what temporal difference learning for the prediction problem would look like in code.
Basically this loop is exactly the same as what we had before except for the fact that instead of updating
v directly we are updating W and B.
So first assume we are given an environment and a policy assume that the policy is a function which
accepts as input a state and returns an action next we initialize the weight and bias to be random.
Next we enter a loop for a predetermined number of episodes inside the loop.
We play one episode we call Ian VDI reset to get the first state s which is now a continuous vector.
As usual we set the Dunn flag to false and enter a loop which exits when done becomes true inside the
loop.
We use our given policy function to get the action a we perform this action in the environment to get
the next state as prime.
The reward R and the done flag.
Then it's time for our big update
in this update.
We'll need the values of both V of s envy of US prime which we can calculate using our linear regression
model then we run our gradient descent step using the gradients we'd arrived earlier.
Lastly we update us to be as prime when the loop is done w and B will have converged
as you'll see very shortly approximating V of us is not that useful for the prediction problem.
We would be given the policy.
But what if we are trying to use the value function to help us choose the action then what is the policy.
Well normally it would involve taking the ARG Max over Q of S and a overall actions but if we have only
V of S this is not indexed by the action a.
This is problematic because how can we figure out what the best action to take is
here's one way we could do it which is not optimal.
Imagine you are in the status and you want to know what is the best action to perform.
Imagine that you are in a very special environment where you can try an action and then go back one
step to put yourself back into the status.
In this case what you can do is try every single action so you arrive at all the possible next states
as prime and receive the reward R then out of all of those actions you choose the one that gives you
the maximum value of our plus gamma times V of US prime.
Of course this approach is not at all ideal because in any realistic environment you cannot simply try
actions and then go back to your original state.
It would certainly be nice if such functionality existed in the real world.
So what do we do instead.
Of course the answer is to use the action value Q In this case.
What is the target as before if as prime is terminal then the return is just are the reward.
If s prime is not terminal then the target is R Plus gamma times the max over all actions a prime of
Q of s prime a prime the prediction is Q of S and a and the parameters we want to update are still w
N.B..
So here's what the update looks like in code.
First the squared error loss is our plus Gamma.
Times the maximum are a prime few of us prime a prime minus Q of s in a all squared the gradient is
Q of S and a minus the target times S.
However an important caveat is that this gradient only applies to the action a which was the action
actually taken since that's where the prediction came from.
The weights for the other actions didn't make predictions.
So they are not updated.
This is of course equivalent to saying that their error is zero which will come into play during the
implementation.
We do the same thing for the bias term only the bias term which influences the prediction for the action
a is updated in other words we index both w N.B. by the action a
finally here's what Q learning looks like when we use approximation methods.
Let's start again with what the choose action function would look like.
Again we use epsilon greedy for exploration First we generate a random number between 0 and 1.
If this number is less than epsilon we perform a random action.
If it's not then we calculate the model outputs for Q Given the state s to choose the action and we
take the ARG Max over Q at the status.
Again this works because the actions are encoded as integers starting from 0
next.
Let's look at the Q learning loop as before we start by initializing W and b to be random.
Then we enter a loop and play one episode per iteration we reset the environment and set the Dunn flag
to false inside the second loop.
We choose an action using the function we defined earlier.
Then we take a step in the environment to arrive in the next state and receive our reward R.
Next is our big update.
It helps the first calculate the target since it looks pretty complicated so we'll call that y it's
equal to our plus gamma times the max over all actions a prime of Q of s primate prime.
And remember AQ itself is a model prediction.
So anytime you see Q here that means we're calling model that predict next.
We run our gradient descent updates for WSB but importantly we only update the components of W and B
which correspond to the chosen action a lastly we update the state s to become as prime.