In this lecture we are going to continue our discussion of deep Q learning and approximation methods.
You'll notice that in the last lecture we didn't invoke the use of deep neural networks at all.
Well first it hopes to do something simple.
Before we do something complicated lay down the foundation before you start adding all the bells and
whistles.
Now you might be wondering is it not trivial to use neural networks for Cuellar.
Is it not the case that all we have to do is replace the linear model with a neural network regression
model.
This is exactly the rule all machine learning interfaces are the same.
So instead of W transpose x plus B we could have a two layer neural network.
As I've written down here this seems perfectly reasonable.
In fact it seems like it would be better than a linear model a linear model can only approximate linear
functions.
How do we know that Q is a linear function of the state s.
In fact it's not clear at all that we can make this assumption.
It doesn't even seem like a good assumption.
Neural networks are more flexible and they can handle all kinds of inputs such as images text time series
and so forth.
So here's what it would look like.
Conceptually our loss is still the same.
It's just the squared error.
Any parameters of the model would be updated using gradient descent.
The only difference now is that the gradient will be different since in this case the gradients are
more complicated.
We don't want to derive them ourselves.
Instead we would like to make use of automatic differentiation which is included in all modern Deep
Learning libraries.
In this case I've written the gradient in terms of a generic parameter theta but you can imagine that
it applies to all the weights and biases of whatever neural network we decide to use.
The problem is training neural networks is inherently unstable.
You probably haven't noticed this yourself.
If you only just started studying deep learning but after a while you'll realize you can't just choose
any learning rate and any optimizer in any model architecture.
Sometimes your cost will explode.
Sometimes your model will not train at all.
Neural networks are pretty sensitive to these choices whereas linear regression usually isn't.
Now if you're a beginner and you're just taking a course on deep learning then the instructor has figured
out all this stuff for you so you don't really recognize the hard work that goes into this
on top of this we have another problem.
Temporal Difference learning itself is also inherently unstable.
Technically we are doing gradient descent but remember that the target is not a real target.
Therefore I lost is not a real loss and our gradient is not a real gradient of a real loss with a real
target.
It's approximations all the way through
what happens when you put together these two inherently unstable algorithms.
Well you get something that barely works if it works at all.
Now that's not to say there won't be instances where you can simply plug in a neuron that work into
your Q learning script and make it work but generally speaking this is not going to work as nicely as
say linear regression for a normal supervised learning dataset
so there are actually a number of approaches called Deep Kulkarni that have been developed over the
years.
But in this lecture my goal is to introduce you to one of these approaches.
The idea is we're going to use something called an experience replay buffer or an experienced replay
memory.
You will notice that in our previous updates we were doing stochastic gradient descent.
In other words looking at one sample at a time what we want to do in order to make this process a little
more stable is Batch gradient descent which looks at multiple samples at a time.
In this way we hope to stabilize the gradient
as a side note in libraries like tensor flow it's called Stochastic gradient descent or SGI.
Even when you use batches however I like to make the distinction that batch gradient descent refers
to looking at multiple samples whereas stochastic gradient descent refers to looking at one sample
you're incurs to try this yourself to gain some intuition grab any supervised learning dataset and compare
the learning curve you get when you look at one sample at a time versus when you look at say thirty
two samples at a time.
You can also compare an entire range of samples from one up to the size of the dataset.
You should find that the lost per iteration is much more stable for batch gradient descent than stochastic
gradient descent.
The other extreme is that you can use the full data set on each round of gradient descent in which case
your last function should decrease Mont atomically
another point is that it's not good to have each sample be very correlated to the previous samples when
you are doing gradient descent.
This is why we like to shuffle the data on each epoch when we are doing gradient descent and reinforcement
learning.
The data is very correlated because you're going from one state to the next in a Markoff process so
you can almost even think of it as like a time series.
Every sample is correlated with the previous sample we'll see how we can randomize the data from the
replay buffer later on
next.
Let's look at how a replay buffer would work.
Conceptually you can imagine it like a python list that stores tuples of the form as they are as prime
and done as we encounter them.
We can call these tuples transitions on this slide.
I visualize what such a list might look like in memory and how we might update it in code.
Basically it's just a one line addition in our loop every time we take a step in the environment.
We just add the latest SARS prime done to our replay list
one important characteristic of the replay buffer is that it has a fixed size at some point.
Some of the transitions in the replay buffer will be old and they will correspond to a policy that we
are no longer following.
In this case we may not want to use these to form batches of data when we perform gradient descent.
So here's a new addition we want to add.
Instead of only appending new transitions to the replay buffer we are also going to remove transitions
when the replay buffer reaches its maximum size.
Since this is a list where we are always appending to the end the transition we want to remove will
always be located at the beginning.
The oldest transition
the last thing we have to consider is what happens when we actually want to perform the update.
In this slide I will describe conceptually what we want to do although in code we will make this a little
more efficient.
So let's suppose we have some function called train in this function.
We first to sample a batch of transitions from our replay buffer.
This is now a list of tuples where each tuple Contains SARS prime done transitions.
Next we initialize our inputs and targets as empty lists.
Then we loop through each transition if done is true.
That means as prime is terminal.
So the return is just ah.
Otherwise the target is our plus gamma times the max over Q of s prime a private after we've created
the target we can append that to our targets list then we add the input the state s to our inputs list
and then once we have a batch of inputs and targets.
This is nothing more than gradient descent.
So all we need to do is the usual gradient descent update parameter minus learning rate times gradient.
So here's what the pseudocode for deep Q learning would look like.
It's very similar to Q learning with approximation methods except now we are going to make use of our
replay buffer.
In fact this code is a little simpler because we already describe the functions to update the replay
buffer and train the model.
So to start we initialize our replay buffer to an empty list.
We also instantiate a model which of course comes with weights which are randomly initialized.
You can obviously picture it as a neuron that we're next we enter a loop that runs for a predetermined
number of episodes inside the loop.
We start a new episode by calling in VDI reset and setting done defaults.
Then we loop through each episode first we choose an action using Epsilon greedy.
Next we take that action in the environment to get s prime R and done.
Then we add the latest transition as a r as prime.
Done to our replay buffer this function also removes the oldest transition if the buffer has reached
its maximum size.
Next we call the train function which samples a batch of data from the replay buffer calculates the
corresponding targets and runs one step of gradient descent.
Lastly we set the state s to the next state as prime for the next iteration of the loop.