-: Hello and welcome back
to the course on artificial intelligence.
In today's tutorial, we're going to have some fun.
We're going to have a look
and artificial intelligence actually going through that maze
that we've been talking about so long.
And is going to use key learning to navigate its way,
and find the way out,
and we'll see what happens to the Q values
what's gonna happen to the policy, and so on.
So, let's have a look.
We're going to be using some materials kindly provided
by the Berkeley University.
So if you go to ai.berkeley b e r k e l e y.edu
if you just go to that link, click enter,
you'll see this website.
And here, what we're going to be looking at is,
you need to go to, where do we need to go to?
PacMan projects, I think.
Yeah, PacMan projects.
And here, if you scroll down and you look
at Reinforcement Learning, this is what we're working with.
So here you can download the ZIP Archive.
So, that's if you want to, so don't,
you don't have to, this is, we're not going to go
through the solution together in this tutorial.
Just letting you know where this is all from
because we are very like, we really appreciate that
UC Berkeley has made these materials available.
But, if you do wish to experiment with this on your own
just bear in mind this is not part, it's not gonna be part
of our course, this is part of the Berkeley course.
I'm just gonna show you how it works
for illustration purposes.
But if you do wanna experiment with this
you can find it here,
the ZIP archive and all the instructions as well.
And we are just going to go into Python right away.
And first thing I wanted to show you is
that here we've got the licensing information.
So, this is what I mean we're very lucky
that they said we are free to use or extend these projects
for educational purposes provided, you know
distribute published solutions, which we're not going to do.
You retain this notice, which we have
and you provide a clear attribution to UC Berkeley
including a link to, which we also have.
So once again, if you'd like to learn more
there's a link you can have a look and thank you very much
to all of these people who have worked on this project.
So, here's the grid world that we're gonna be working with.
There's a solution there.
You would have to, in order to make it work
you'd have to either sold it yourself
or possibly find a solution.
Maybe some of your, some people,
somebody you know might help you out of that.
If again, you want to, you don't have to
because we are just going to look
at it on this screen right now.
So, after we've created all those files,
we can just launch it over here.
So, there are some parameters that are involved
in this whole world.
And now I'm going to just show you how,
what it looks like if we launch it.
So, let's try launch it in manual mode.
So, if I go minus M one of these parameters here manual,
so I can manually control the agent.
So here you can see how grid, so I can go up, up,
so you can see that it's taking action,
starting and started in state.
So where I was, and then you saw
that I pressed up, took action north
and first time I ended up in 01, so I did go up.
But second time I took action north,
and I ended in the same, so I didn't move.
So something happened, you know, the randomness happened
I either went left or right,
and by default the parameters are set.
You can see here by default they're set
to exactly what we discussed,
that how often action results
in unintended direction 20% of the time,
10% to the left, 10% to the right.
So if I go up
you see I went up, I go right, I went right, right,
no, didn't happen right again and right, and I'm finished.
But in this implementation,
you have to click again to get out of this final output.
So out of the exit, just click again and you're finished.
That's a terminal state.
So, we can run again, manual.
You can see that if I go right, right, right, left, up.
So here what we saw previously
that the agent wouldn't go straight up, right?
What's the point of going up
if there's a chance of going into the pit.
So let's see, what would the agent would do?
It would go left, you'd go west here.
So it'd go west and you see I clicked left, but it went up.
And here I would click right,
and I end up in the final exit sync.
And you can see it got reward equal to one.
So that's what it looks like manually.
Now, let's actually hook up an AI
to this and let it go through.
So let's do an H here and let's add some parameters.
So, let me just see what I typed here.
So hopefully you can see Python, gridworld.pui
then here, minus R means that's the reward for living.
So I've got two of them,
so I probably should remove this one.
So minus K is how many iterations?
That's way too many iterations.
Let's do less, Let's do like 10 iterations,
that should be enough.
Minus A is agent.
What type of agent do I want,
do I want some random agent, some value agent, or a Q?
Q, so by, I want a Q learning agent doing this.
Minus S is, what is S?
Speed.
So that's way too fast, lets just use default speed for now.
Minus R is the living penalty, so by default is zero.
So remember at the very start we started
with zero living penalty.
So let's call it also zero here,
we can just remove this parameter.
And D is, what is D?
Discount.
So our discount factor, so let's keep it at 0.9.
So very similar to what we were starting
off in this section of the course,
so let's run that.
Okay, way too fast again, I think.
Oh, actually it's pretty okay.
So you can see how he's exploring.
And so, so far he's hit the negative three times,
and you can see how the Q values have been updated
in these squares.
So these are Q values, they started with zero.
You can see now the Q values,
so he's learned that again,
this one is a bit different implemented
because once you get to the final stage,
you have to get out of it.
You have to just click one more button to exit,
and so it's very close to one, but not exactly one.
But at the same time, you can see that here
you know, the value is slowly kind of crystallizing in 0.8.
It's slowly getting somewhere,
but the rest so far they're kind of zeros
because he doesn't have enough information
to understand what's going on.
Okay, so let's see what happens here.
Exploring, exploring, exploring.
What's gonna happen?
Whoa, it's been a while.
And don't forget this, some randomness involved here.
So there's, he hit that good one a few times now.
He only gets 10 iterations, so he's gotta learn fast.
Okay, need you there, let's see what's going on.
Come on, get out of that maze already.
And yes, 10 episodes.
So, average returns, that doesn't, that's,
we're not really interested in that.
So here, let's see, I've never seen them before.
I click right, there we go,
so you can see this is the policy that he came up with.
Even through just 10 episodes, he's already got a policy.
Okay, I'm gonna go "buh, buh, buh, bum",
and here I'm gonna go down,
here I'm gonna go down,
here I'm gonna go into the wall,
and then I'm gonna bounce over here.
That's pretty cool.
Okay, so now let's increase the speed.
What was the parameter S there?
And let's like double, well, let's quadruple the speed,
and let's increase the number of iterations.
So, let's say 20 iterations this time.
And let's see if he can get through a bit more now.
So you can see he's going a bit faster,
and he's learning, he's learning that it's not really,
you know, out of this state that's,
there's not many good actions
or you know these actions that right
and straight are not that good.
Definitely this one's definitely not good.
He still needs to learn that.
So from here it's also not good.
You can see that this action's pretty good.
All right, what did he get?
Okay, so interesting policy here.
He decided to go up, just not enough information.
So let's redo that,
and let's increase the speed to like a hundred.
So it's super fast and the number of iterations,
we'll give him a hundred iterations this time.
Let's run that.
See he's like crazy fast.
And you can see that
because there's so many more iterations,
he's got more information, more opportunity to experiment
and to actually build out this matrix,
or not matrix, these Q values
for every single state he now knows.
Okay, you can see that 0.89,
what did we say in our example is like 0.86?
Another thing to remember this that the value
of any given state, remember that the formula we had,
is the maximum of the Q values.
Remember that thing that we came up with, shortcut formula.
So what is, what would the value in this state be?
The V of this state?
It would be 0.89
cause that's the highest out of the four.
Here, the value of this state is 0.71.
The value of this state is 0.61, and so on.
So that's something to remember.
So, and, remember in our example
I think we had like 0.86 or something, so pretty close.
And so, if we go next here,
oh, it just disappeared, where did it disappear?
Let's do it again, and let's make it come back.
Okay, okay, slowly, slowly, slowly filling up some spaces.
I see, and it's also pretty random
because not only the environment has randomness,
but also the way he explores at the star
when he doesn't know the policy is he's exploring at random.
Just keeps disappearing, I don't understand why.
Anyways, so let's see what happens
if we increase the number here and here.
Should pretty much take the same amount of time
if the speed doesn't have a cap on it.
Okay, so you can see it's like,
it has more opportunity to explore things.
Okay, let's see how it all goes.
And you can see the values are converging,
they go up and down depending, you know,
because there's some randomness and he might end up like
in the pit even though he goes like this way.
But at the same time they're slowly starting to converge
to some sort of end values and Q values.
Okay, probably a thousand is a bit too much
in terms of time.
It doesn't look like the speed
is proportionally increasing as well.
So, it might cut that part.
I mean, like, reduce the speed.
Yeah, wow, this is very long.
You don't have to watch through the end of this tutorial.
I just wanna experiment with quite a bit.
So, to give you some
like examples of what we've been working through.
But you get the point that it goes through all of this,
it has some randomness,
like, randomness built into its behavior.
So even when it has like a policy,
it will still continue exploring.
So it won't just like once it has the basic policy
it won't just continue following its policy.
It will still experiment with other variations once
in a while in order to enhance its policy.
Maybe it hasn't found the best policy already right away.
Maybe it can improve the policy,
and that's why even after so many iterations
you can still see some random effects.
It sometimes jumps into random states,
not just because of the randomness in the environment,
but also because there is some level, like a parameter
which you could control, which you could set up
for your agent saying that you know, most of the time,
80% of the time do whatever your policy tells you to do,
but 20% of the time, you know, just have some fun,
experiment, and see what happens.
And, use that information that you gather
to update your policy.
Okay, this is taking way too long.
Let's try that again.
Yeah, so that's how the agent learns in different states.
Maybe let's just run one more just out of curiosity.
So, is there anything else we can change about iterations?
Duh, duh, duh, okay, okay, let's have a look.
Yeah, well we could change the discount for example.
So, in this case we could say K minus 100
minus A Q minus S to the minus R.
Okay, 2000.
So reward, we want to keep it, maybe let's keep it at 0.04,
but let's say, say it again,
let's keep the reward at minus 0.04 every time.
And then here we're going to say that
D, the discount is not 0.9 but it's like 0.5.
So, it gets discounted quite a lot
as you go through the game.
So it actually, now it'll be incentivized to be closer
to the finish rather than further.
Well, the state's closer to finish will get high value.
So you can see that the values it quickly drops off.
It's not as as green as it was before.
So here you can see that this is the policy now.
So it goes like that, like that, like that, like that.
Very similar to what we saw before.
Just probably the only differences
from here is jumping straight into here.
So that's that one.
And okay, let's just run one more, this is so much fun.
Let's just run one more.
So, K minus K 100, AQ, discount,
keep it as it was, original.
So like just run this basic vanilla setup.
Okay, okay, okay, it's going,
let's see if it will show us the policy at the end.
Yes, we got the policy.
Yes, good finish.
So here we've got the policy.
You know, this is familiar.
Remember that time when we saw that the AI outsmarted
the human went boom into the wall to go there and boom
into the wall to go like that, to increase the probability.
So there we go.
That is an example of Artificial Intelligence in action.
Very, very basic, simple, key learning.
So no deep learning at this stage,
but at the same time it's already pretty smart.
And I hope you enjoyed today's tutorial,
and once again thank you to UC Berkeley,
and I hope you enjoyed today's tutorial,
and I look forward to seeing you next time.
Until then, enjoy AI.