-: Hello and welcome back
to the course on Artificial Intelligence.
Alright, so I hope you're enjoying the tutorial so far.
We're nearly done with the intuition.
You'll soon, very soon get to the practical side of things.
We've just got a few little things that we need
to cover off.
Alright, so previously we talked
about how we add neural networks into this whole equation
of Q-Learning and take Q-Learning
to the next step and turn it into deep Q-Learning.
And today we're going to add
an extra important feature which you will be coding
in the practical side of things.
So Hadlan and I decided
that it's important for us to cover it off
in the intuition side of things so that you're more prepared
for it when it comes in the coding side of things.
So as we discussed, we've got the network there.
There's two parts that happen.
First of all, it's the learning.
So the network actually learns with every new state.
It slowly updates its weights to get better
and better and better at dealing with this environment.
And then there is the acting inside the state.
So after the Q-values have been counted in the state
then one Q-value selected.
So today we're still going to talk about the learning part.
We're going to come up
with a interesting feature that's going to, well,
we're not going to come up with this feature ourselves,
but we'll talk about a feature that is very important
for deep Q-Learning
and that feature is called Experience Replay.
Alright, so here is our network.
So we've just copied it over here.
We've got that loss that is calculated
at the bottom is back propagated through our network.
And let's have a look at an example
of what happens to understand the problem that we're dealing
with a bit better.
So here is an example actually from this course.
This is a screenshot exactly from this course.
This is what you'll be programming.
This is a self-driving car,
that is driving through this, along this road.
And it has to learn how to navigate this road.
And so what is, as we discussed previously,
what is this state?
And of course the state is not just gonna be x1 and x2.
Hadlan will describe a bit
in a lot more detail what the state is.
It is gonna be a couple of parameters which relate
to the angle of the car and some relative parameters,
what the sensors are reading and so on.
So there's gonna be more parameters than that
to describe the state,
but nevertheless, there's going to be a vector of values.
It's gonna go through a neural network.
And then on the output, you're gonna have some Q-values.
Again, there'll be a different,
depending on the environment,
there can be a different number of actions,
possible actions.
But we're just going to, for simplicity's sake,
leave it at four just for us to be able
to understand a bit better what's going on here.
So in this case,
the question is so far,
what is this input into this neural network,
or more specifically,
how often do we trigger this neural network?
How often does this neural network go through?
Well, every time the car ends up in a new state,
so the car makes a move, it ends up in a new state,
and then everything goes, all that data,
all that information from about the state goes
through the network,
Q-values are calculated,
this error is calculated,
based on what we discussed in previous tutorials.
This error is back propagated through a network,
weights are updated,
then the car selects which action wants to take,
makes that move, ends up in a new state,
in the new state, everything starts over again.
And so basically this happens every time the car is
in a new state.
Well have a look at this example.
I specifically took this screenshot because it looks,
it's very well illustrates the problem that is addressed
through Experience Replay.
And Experience Replay is not just something that we use
in this course or in this specific problem.
It is something that you will see used throughout,
like on and on and over and over again,
in artificial intelligence algorithms,
because it is so powerful and it's so important.
So look at this car,
this car in this problem or in this environment,
its goal is to go from here to here and back.
Its goal is to navigate its way here to here
without crossing these walls, which are made of sand.
And so the car start over here,
it went down and like its reward is based on, you know,
how close it is to start.
So the car went from here, it went down,
and kept going like this, like this, like this, like this.
Along this wall, along this wall.
And what is going to do next is going to turn,
it's going to keep going.
Well, what you want it to do is keep going here.
But let's think about it for a second.
Once it got to this wall,
every single time it moves forward,
it moves forward, it moves forward,
it moves forward, it moves forward, it moves forward
it moves forward and so on, it moves forward.
So there might be like,
depending on the structure of environment,
it could be like 100 moves here, or 50 moves here,
that just keeps moving forward,
forward, forward, forward, forward, forward.
And nothing changes, nothing really changes.
Yes, it gets further away from this target,
closer to this target, that's lovely,
but in terms of the surrounding environment,
not many things are changing.
It's still that same wall.
If you are sitting in the car,
you've probably seen this situation,
when you're driving, whatever you're seeing,
is like the environment is so monotonous,
that you're just seeing kind of the same thing
as just passing by.
But like imagine you're driving through a desert
and you're just seeing the same thing,
it's the same sand, it's the same sand,
nothing is happening, nothing is changing.
And so every single time we're putting that state,
that new state, into here.
Yes, of course, something might be changing.
For instance, you're driving the car,
and your GPS is showing you're closer to your destination.
So one of these inputs is changing,
but a lot of these other inputs,
the sensors for instance which are on the car,
they're not changing.
And therefore as you're driving,
so in this state you input the inputs
into your neural network here,
here, here, here, here, here, here, here and here,
all the time, the inputs are pretty much the same.
And so if you keep inputting the same inputs,
the same values, the same vector,
or very similar vectors into your network,
because there is no variety,
the car will learn very well one thing.
It'll learn very well how to drive along this wall,
which is on its right.
And so that's how the network will update,
and it will get rewarded,
will slowly start getting rewarded for driving so well.
It'll be like, oh, okay.
So from here it'll be start learning,
"Oh, I'm doing so good, I'm doing even better.
I'm doing it better."
It will have this false perception
that it's actually doing very well,
even though it only learned how to drive along this wall.
And so the neural network will become very adapted
to driving along this wall.
And then all of a sudden there's this curve
and the car doesn't know what to do.
And it completely doesn't fit in with this neural network.
And even if it does adjust, somehow,
let's hypothetically say it passes this part,
and then it ends up on this wall,
same thing is gonna happen,
it's gonna drive from here, here, here.
Okay. Now the neural network is restructuring itself
to adapt to this wall, and then bam, this thing happens.
And then even if somehow it gets past that,
it'll drive past this thing.
And then same thing along these lines.
So basically it's like a very vivid example
of the problem that we have is that,
because the way we're using the neural network,
updating it with every single state,
once we have lots of consecutive states,
they don't even have to be the same,
but there is, in environments it's normal
that consecutive states are somehow correlated,
or somehow interdependent.
And we don't want that interdependency to bias our network.
We don't want the car to just learn how to drive along
like a straight line or along curved line,
or like anything that you can think of in life
where an agent would be navigating environment
wherever you can think of correlated
or interdependent states that come after another,
that can really mess up your neural network,
if you're just going to let the agent learn from that.
And that's where Experience Replay comes in.
What happens in Experience Replay is,
these experiences, so these states that it's in
one, two, three, however many 50 states here in a row,
they don't get put through the network right away.
They're actually saved into memory of the agent.
And so for instance, it saves all these,
and saves all these,
and at some point,
once it reaches a certain threshold,
which you'll be able to code
and Hadlan will show how to do that,
once it reaches a certain threshold,
then the agent decides for itself,
"Okay, it's time to learn."
I have this batch of experiences that I have
and now I'm going to learn from that batch.
And so it randomly selects a uniformly distributed
and uniformly is key is important here
because that's something we'll talk about on the next slide.
We'll mention that,
but it takes a uniformly distributed sample,
so basically all experiences are considered to be equal.
It takes a uniformly distributed sample
from that batch of experiences that it has
and then it goes through them and it learns from them.
So it doesn't take all the experiences
it just takes a uniformly distributed sample.
So it might take a couple from here,
couple from here, couple from here.
And each experience is characterized by the state
it was in, the action that it took,
the state it ended up in, and the reward
it achieved through that action in that specific state.
So four elements in each experience,
state one, actions, state two and reward.
And so it takes all those experiences
and then it passes them through the network and it learns,
and that way it breaks the pattern of that bias
which comes from the sequential nature
of the experiences if you were to put them
through the network one after the other.
So that's the main focus of Experience Replay.
That's the problem it addresses.
And another benefit of experience replay is
that sometimes in an environment like this,
you might have very valuable rare experiences.
So for instance, I don't know, let's say,
let's look at this corner, right?
This is a right corner, right?
And a very sharp one.
How many sharp?
So it'll be coming from here,
assuming it's going to be hugging this corner.
So how many sharp right corners do we have
in this whole environment?
We only have one right corner here,
and one right corner here,
Right? So when it's coming this way,
that's the right corner.
And then when it's going back,
it's a sharp right corner here.
So, and this one's not sharp, this one is sharp.
So there's only one opportunity
in the whole environment to learn from a sharp right corner.
And that's an important experience
because it might get really good
at driving along straight lines,
get really good at doing like soft corners
like that, like that,
but then it'll keep messing up
this sharp right corner simply
because it doesn't have that much opportunity
to learn from it.
And so therefore it'll learn everything else very quickly
but it'll take a long time
to learn this right corner.
It's a very simplified example,
it's a very simplified explanation,
but it illustrates the concept
that sometimes there are rare experiences
which can be valuable.
And if you are just doing a simple neural network
where you're putting in your values here,
and you know they're going through, and you know,
like even if we forget about that problem
of the sequential nature of experiences
and how they can be interdependent or correlated,
even if we forget about that for a second,
what happens is once you put an experience in,
it goes through, network's updated,
then you instantly forget about that experience
and you move on to the next one.
That's just how the neural network works.
Then you move on to the next state, the next state,
the next state, the next experience, the next experience,
the next experience and so on.
So this right corner,
as soon as it goes through the network, it's gone.
And you don't have any memory of that valuable experience.
Whereas with Experienced Replay,
because you're putting these experiences into batches,
you can organize your batch as a rolling window.
So for instance, you could have like 100 batches,
so 100 experiences in your batch,
so when it's coming back from here,
as soon as it has recorded this experience in its batch,
then like at some point it takes a uniform distribution
from its batch of experiences,
and then there's a rolling window.
So it forgets these experiences,
but then it keeps these experiences,
and then again it learns from, once it's here,
it learns from this batch.
And then once it's here, it forgets all the way up to here.
But then it has a batch of experiences like that.
So therefore now it learns from these experiences.
And that way what you're getting is that
this right hand corner might come up several times
in its learning process because it was in that batch
when the batch was like this,
around there, then it was in the batch here,
in the batch here.
So it came up in several batches,
because a batch might be updated
as a rolling window of experiences.
So the older experiences get kicked out,
the newer experiences are added,
and then again, older experience kicked out,
so an experience,
it stays in the batch for quite some time
and the car or agent can learn
from that experience several times.
So that's another advantage of Experience Replay.
And of course the final advantage is
Experience Replay gives you an opportunity to learn
from more experiences than if you were just learning
for one at a time,
because you have that batch
and therefore, and it's a rolling window,
and therefore, even if your environment is limited
to experience, your Experience Replay approach
can help you learn faster.
And instead of just redoing their environment,
many many times, you can learn faster
because you you don't have to redo it.
You have those experiences saved.
So those are the main advantages of Experience Replay.
Let's recap on then we've got the,
we're breaking the pattern of independence
and correlation of sequential experiences.
We save rare experiences, which might be important,
and therefore we can learn from them more often.
And we can learn in environments,
we can learn foster environments which are experience,
which have a shortage of experiences
or which don't have that many experiences
that the agent goes through.
And still we can be able to learn that.
So that is what Experience Replay is all about.
If you'd like to read a bit more,
there's an interesting article published
by Deep Mind in 2016.
It's called "Prioritized Experience Replay".
And it talks about
why are we using a uniform distribution
to select our experiences from the experienced batch,
why don't we find a better way to select our experiences
and prioritize some of the experiences
which we feel that are important.
And so it's quite an interesting thing.
So in this case, you will be able to not only reinforce,
not only reinforce your knowledge on Experience Replay,
but you'll actually be able to move
with the cutting edge of technology.
So this is 2016 and published by Deep Mind.
So it's a very recent, very powerful paper.
So you'll be able to actually explore the limits,
or explore even further this algorithm,
and take it to the next level.
So I'll leave it up to you to find out
why and how we can change the uniform distribution
to a different approach to Experience Replay
from this paper if you'd like.
And I hope you enjoy today's tutorial,
and now we know what Experience Replay is
and we can confidently use it in our practical tutorials.
And I look forward to seeing you next time.
Until then, enjoy AI.