Instructor: Hello and welcome back
to the course on artificial intelligence.
Previously we had quite a strenuous and long tutorial
on market decision processes
and hopefully you got along well with that
and hopefully I could explain things
in a approachable and engaging way.
And today we're going to talk about policies versus plans.
It's gonna be a quick and fun tutorial
because now we're entering into a new world.
We're entering into a world of stochastic search,
not deterministic search.
When you, it's not just about getting through the maze
but also accounting
for random factors that might just hit you
in the head when you're going through this maze.
And you need to be prepared for them.
That's the world our agent is living in.
And it's more fun, but it's also more dangerous.
It's more, it's less predictable.
So how is our agent going to behave?
Let's have a look.
There's our mark of decision process framework
which is once again our favorite bellman equation.
However, the more advanced version of the bellman equation
we were working with.
So from now on
we're just gonna call this the bellman equation.
And here we've got our maximum across all actions.
So the value of a state, any S state S is the maximum
across all actions that an agent can possibly perform
in that state.
And the maximum was taken from the reward
that the agent will get by perform action A
in state S plus a discount factor multiplied
by the expected value of the new state it will be in.
And the expected value is taken here because there are
it doesn't know exactly what state will end up in.
There are some random effects that are present
in the environment that might alter the state
and you might not end up in the desired state.
It might up end up in a different state.
That's why we're taking the expected value over here.
This sum over here.
So let's have a look at this
as our example or in our example of the maze.
So this is what we had previously.
So previously we're dealing with
we've deterministic search.
So we knew that. All right, so if I'm here
I definitely need to go here.
If I'm here, I definitely need to go here.
If I'm here, I'm here.
So it was all pretty straightforward once you have this map
and remember we called it a,
we called it a plan.
Once you have the plan, it's pretty straightforward,
you need to do.
there are our arrows.
So that's the plan with arrows.
And from here it was very straightforward.
We're, this is, these are the routes
that the agent would take
wherever you start on this blue line.
That's exactly the way you would go.
However, now we don't have a plan anymore.
We can't have a plan because you know,
whatever we plan might not happen.
It's not under a control.
A plan is when you know exactly what you need to do next.
You know the steps.
So you have a starting point, you have a goal
and you know every single step.
So you can plan them out.
You're like, I'll do this one, I'll do this one
I'll do this one.
Like your life, like a plan.
But at the same time,
there's so much now randomness going on,
you can't have a plan
because what if you get here and then you click to
the right and actually takes you down.
So that's not part of your plan.
So that's why it's not called a plan anymore.
And here we're going to calculate the values
or we're actually going to just look at
the calculated values for this same problem,
but based on based
or given that we have this randomness inside.
So these are the new values.
And so why are these values different?
So let's just compare to what we had previously.
This is what we had previously.
These are the new values.
So once again, we had previously, you can see 1, 0.9
0.81, 73, 66, and this is what we have now, 86.
So less than 1,
74, 71, 6, 3.
And so, and by the way, these are not exactly
the correct values are off the top of my head.
But if we were to run an agent,
values would be something similar to this.
And the values could change because depending
on the gamma that we choose, 0.9 or other value.
But nevertheless, for arguments sake,
these are the values that we're dealing with now.
And there approximate,
they convey the whole notion in the correct way.
So let's have a look at them.
Why have they changed?
Well, why is here, Let's start with this one here.
The value was one, why is it all of a sudden
in 0.86, why is it less than one?
Can't we just go from here?
Well, we actually call it, because from here
if we go right, which is our intention
if we go right, we would actually
with a 10% chance, we'd end up here.
So we'd hit the wall and we'd be back in this state.
And remember, we have a gamma
so the value would be discounted.
And or with a 10% chance,
we would end up here in this state.
So it's not a hundred percent probability I would get here.
So therefore this value can no longer be a one.
It's something less.
And it's, let's say is 0.86.
So that's an example of why it's like this.
And you could get the exact value
if you calculated the bellman equation
the full bellman equation that we have.
Now, the only problem is that there's
there'd be some recursion
because you would need to know the value
for this and then you would need to know the value for this.
It's quite complex
and that's why we're not doing
the calculations manually here.
That's why the AI, but the AI can do them
as it's going through, through all this.
It's like nothing too complex
for the AI to calculate these things.
So that's our value here.
But now, let's take a look at different one.
So here it used to be 0.9 just
because the discounting factory.
Remember from here to here again, now from here
we can't just jump from here to here,
simply because even if we jump, if we go like this,
we might end up back here, back here, right?
There's 20% chance that we'll still stay
in the square because we'll hit a wall and again and so on.
So the value of being here is 0.71.
Again, this the discounting factor.
You know, this might look odd to you that this is even
with the discounting factor, this is too high.
Maybe the discounting factor in this example is not 0.9
maybe it's 0.99 or something like that.
So don't worry about that.
Just kind of like focus on
that the values have indeed changed
that the values are now less
mostly because it's not a hundred percent probability
to get to the state that you want to get.
And what you will find an interesting one is here
that here it used to be 0.9
and it actually has dropped very much.
It's dropped substantially.
Why is that?
Well, because if you go from here up, which is our intention
there's a 10% chance of hitting the wall
but there's a 10% chance of actually ending up
in the fire pit and losing a minus one to reward.
And basically that means for the agent,
that's end of the game.
And so this is a very bad state to be in.
So all of a sudden,
remember we had 0.9 here, zero point (indistinct).
So they were equivalent.
It doesn't matter you're here
or here they're pretty much equal
in terms of value of being in each of these states.
But now all of a sudden, bam.
This state is like nearly twice as good as this one.
Simply just because here, if you go straight to
you go right where you want to go.
The, you know, the consequences
of the randomness occurring is you just stay here.
Here, the one of the consequences
the 10% chance is you end up in the pit.
So as you can see,
this is no longer such a good state anymore.
Simply because of something that fluctuation
that could happen.
As you can see, this one is also very bad
because it's as bad as this one in terms of, you know,
it's only 10% chance of ending up in the pit
and 10% chance of ending up in the wall.
But at the same time, there's the discounting factor.
So first of all, the discounting factor.
And also after this one, you'd have to go here
and even if you hypothetically went here,
you could end up in the pit again.
So that chance would also be taking into account
because remember this value is derived from this value
and this value is derived from this value, right?
And therefore it's small.
But in reality, actually that what I said there was wrong.
This value is not derive from this value.
So if you just have a look now,
you will notice that this value via here is actually
greater than this one.
You will notice that for the agent
it's better to go around this way than this way.
And it makes sense, right?
Because this way it doesn't lose,
there's no chance of getting in the pit.
Yes it is a bit longer,
and therefore the discounting factor has a bigger effect.
But at the same time,
simply because there's a chance of getting in the pit here,
if it goes straight, it will
there's a chance of it jumping into the pit.
So it'll take, it'd rather take its time
and it'll just like go around
because that way
there's a much lesser chance of it getting into the pit.
There still is.
So if from here it goes there, from here, it goes there
it could potentially get into the pit
because it could end up there
and then it could end up in the pit,
but nevertheless, it's a lesser chance.
So it would just go around like that.
So very interesting to see how they all changed.
Remember previously from here you would go like that.
From here you'd go like that
and from here you go like that.
And now all of a sudden you can see it's changed.
So let's draw the arrows and see what it looks like now.
And overall, yeah
you see even more random thing, right?
So yes, this is true, but look at what happened here.
Look at this one, look at this one.
Were you expecting that?
That's something definitely like
when I saw this for the first time, I was very impressed.
I was not,
I was surprised and I was not expecting this at all.
And this is, this is an example of, you know
when AI can outsmart a human.
It's like something you can't even, you could
I can't even predict, but the AI
through reinforcement learning,
remember that example of the dogs
that can actually sometimes work better
than normal real life dogs or pre-programmed robot dogs
or can play soccer,
simply because they come up with these ideas
that even we can't see.
And so that's a great example, right?
So you probably weren't expecting that as well.
That the agent instead of going up is like,
why would I, like, if I go up
then there's a 10% chance I'll jump into the pit
but what is it achieved by going into the wall?
Well, 80% of the time it'll bump back and stay in this state
but 10% of the time it'll go here
and 10% of the time it'll go here.
So all of a sudden you can see that now it's actually
in this new approach of jumping into the wall
there is a 0% chance it will go into
the fire pit from this spot.
So, and it's like
it really doesn't want to go into the fire pit.
So it rather bounces into the wall
a couple of times and then it will go either right or left
at some point because that randomness is gonna happen.
And so it learned that through experimentation
it learned that, okay, when I go forward
the results are not as good as when I go to the wall.
And if you think about it, it's like this, this robot
if you think about it, like there's a fire pit
there's a very, this is the, this is like a, this square is
like a very tiny ledge
and then this is like a mountain like a cliff,
and this robot is just hugging the cliff
and just, you know, like trying to waiting
until it like pushes it right or left.
Because, well, as a human, you'd probably do the same.
You wouldn't be standing facing that way
that way you'd be hugging the the cliff, right?
Or something like that.
And hopefully you never need to end up in a
you never end up in situation like that.
But like visually, just visually
if you think about it, same thing here.
And so that's pretty intense, right?
So that, that the AI came up with this idea.
And same here, that instead of going left
and risking getting a fire,
I'm just gonna try balance off the wall.
Like, you know, hug the wall, try to jump into the wall.
And at some point I know that, you know
just there's a probability
there's a 10% chance every time I do that I'll go here
and sometimes it'll happen and I'll end up here
and I'll be safe and then I'll just keep going like that.
So very, very interesting approach that the AI took here.
And as you can see the, the routes are like this.
So from here it might go right and then it'll go right
to the exit or here it'll go left like that.
And here it will at some point it will go left
and it'll go like that again,
this is important to understand it's not a policy.
So even when it jumps from here, it will go here maybe.
And then from here it might actually,
instead of going straight,
it might actually go back to the right
and then from here it might go to the left,
it might go to the right.
So there's lots of different options for it to go.
So it might not follow exactly the siren go the other way.
This is just the desired routes
that it's designed for itself.
But the way it'll work out is actually could be different.
It depends on the real world.
So there we go.
That's the world of artificial intelligence.
That's what a policy versus a plan is.
And hopefully you're getting slowly
getting excited by what the AI can do,
especially given what we saw over here.
These are some very virtual also type of decisions
that the AI is coming up with.
And as you can see when you apply AI,
even from this small example, you can see that
even when you apply AI in real world
maybe you'll come up with ideas and decisions
that even sometimes humans can't come up with.
And that's exactly kind of like what happened
in those games where the Google AlphaGo was playing
versus Lee Sedol champion of Go in Korea back in, you know
the world champion of Go.
And they were playing in Korea back, back in 2016
I think it was March, 2016,
it came up with some moves that humans had never played
in 3000 years or humans were not used to playing.
And this is exactly an example of that.
So once again, hope you are getting excited
and pumped about this course
and about what we're going to create
and I look forward to seeing you next time.
Until then, enjoy AI.