-: Hello and welcome back to the course
on artificial intelligence.
Today we're talking about the living penalty.
All right, so here we've got our Bellman equation
and as we've been going through this course
we've been slowly making more and more complex.
So so far we've already added these probabilities
in here and also we've added the discounting factor.
Now we're going to look in more detail
at this side of the equation where we have the reward.
Now remember previously
when we talked about how reinforcement learning works
we said we have an agent
and it performs actions in the environment and
in exchange or as a result of that, it gets a new state
in which it's now in and a reward for that action.
Well, so far in our example
we've only been getting rewards at the very end.
Either if we get to the finish line
or if we for the agent ends up in the fire pit
he gets a plus one or a minus one reward.
But that is a very simplistic approach
to reinforcement learning.
And in more realistic scenarios
you will likely have rewards throughout the journey
not just at the very end.
You might have rewards throughout the journey.
For instance, if it's an AI playing a game
and if for example, it's like shooting somebody in Doom
it might get points for killing that enemy.
Or it might be in a different other game
if it overtakes another car or something like that.
Just because of the rules of the game
not because of its way of analyzing the game
but actually the game is structured
in a way that it's reinforcing, it's giving points
for doing certain actions even before the game is over.
So scenarios like that are very common
not just in games and also in real life.
And that's why we're going to introduce something similar
into our example, a simplified version of that
but nevertheless, a reward that is continuously given
to the agent throughout the game, not at just at the end.
And the way we're going to do it
is by looking at the other tiles.
So right now we only have reward plus one at the final tile
and reward minus one at the other final tile, the fire pit.
But now we're going to add rewards in every single tile.
We'll add a very small reward, it'll be minus 0.04,
and as you can see, it's negative.
So every time the agent moves, he'll get a negative reward.
And that's why it's called a living penalty.
Because no matter where he goes, he will always
get this negative reward except for these final tiles
because that's the end of the game.
And so here you can see the reward
even on this tile is minus 0.04.
But that doesn't mean that he starts with that reward.
He only gets this reward, and this is important to him,
remember, he only gets this reward when he enters a tile.
So whenever he promises an action, he goes here
then he will get this reward minus 0.04,
and then if he comes back to this tile, he'll get another
minus 0.04 reward.
And so the longer he walks around, the more
he accumulates this negative reward,
and therefore it is an incentive for him
to finish the game earlier as quickly as possible.
And so now let's have a look at how
our policy or how the agent's policy is going
to change depending on what value we set for this reward.
So here are four environments,
and in each one we're going to explore a different reward.
Now, we're not going to do the calculations
we're just going to project results.
And you will see that intuitively they make total sense.
So here we've got a reward for any step
or for any for getting into any state is equal to zero,
just as what we've seen before here, the reward is going
to be minus 0.04, what we just introduced now.
Here, the reward will be minus 0.5,
or the living penalty will be minus 0.5.
So much higher you can see than here,
more than 10 times greater.
And here the living penalty will be minus two.
So even more than the reward you get
for jumping or even less than the reward
that the agent gets for ending up in the fire pit.
So let's have a look at how the actions
or the optimal policy for passing this environment
will change depending on this reward.
So this is our original policy, and as you can remember
we had these two very interesting
and even a little bit weird decisions
by the agent but which totally makes sense
if he can live for as long as he likes,
if he can just travel around for as long as he wants
without being penalized for, staying alive very long,
why wouldn't he just go into the corner here
into the wall and just keep doing that until it happens,
it so happens that he goes this way
and then he will walk around?
And same thing here, it's much safer for him
to jump into the wall hoping that one of these will come up
eventually, and then he'll go to the finish line anyway
because in the, by choosing these two actions, he
doesn't risk getting into the fire pit.
Now let's see what happens if we add a reward,
a negative reward for just being alive,
for making a step, making a move.
So here you can see that instantly these two changed.
Now the agent doesn't want to jump into the wall.
He is more likely to risk getting to the fire pit
having a 10% chance of jumping in here, but he
will go forward because every time he jumps
into the wall here, if he was going to be doing it here
as well, every time he jumps
into wall he performs an action he ends up
into in this state with an 80% chance.
And that means with an 80% chance, he'll get a minus 0.04
reward, meaning that a lot of the time he's going
to be getting this accumulating, this negative reward.
Same thing here.
If he jumps into the wall waiting for that moment
when he will actually be randomly moved to the right
if he keeps doing that, he will accumulate this
negative reward and that, the result
of that if you perform the calculations, you'll see
that the result of that, the expected value, like
of that approach jumping to the wall,
is worse than taking the risk
of going forward and actually ending up in the fire pit.
So he changes his decisions
in these two blocks to instead move forward and here move
to the left even though there's a risk of jumping
in the fire pit, simply because now, the longer he's alive
the longer accumulate this living penalty
in the next environment.
Now we're increasing the living penalty
to even a greater number minus 0.5.
And let's see what changes here.
So now you can see that compared
to this environment the only thing that changed here
is that this arrow is pointing to the right.
And what that means is
that now it's no longer a good option for the agent.
Oh, actually also this arrow is pointing
was pointing to the left
and now it's pointing upwards.
So now it's no longer a good idea
for the agent to go around from here, go around all the way
because if he goes around all the way, yes, he's safer
there's a lesser chance, there's no chance
of getting to the fire pit, but at the same time
or there's a less chance of getting to the fire.
But at the same time
he will accumulate quite a substantial negative reward
as he walks around.
So it's just, it's the path is too long.
So that forces him, even whether he's here
or here to take the shorter route
to get here even though he has a much higher risk
of getting into the fire pit, because
as soon as he ends up in the square, there's a 10% chance
of getting to the fire pit.
According to his calculations,
it's just the expected value of this approach is better
than the expected value of going
around simply because we've increased this living penalty.
And finally, we are getting
to the example with the living penalty of minus 2.0.
So here I would encourage you to pause the video
now that you've seen how the policy has changed
as we increased the living penalty.
I encourage you to pause the video
and to think for yourself what will happen in this scenario?
What do you think the optimal policy will be given
that the living penalty is so high?
So I'll let you pause the video if you'd like to.
And now I'm going to jump into showing you the solution.
So in this case, if you increase the P penalty
to minus 2.0 it's so high.
Remember that the penalty here is only minus 1.0.
It's so high that the agent just wants
to get out of the game in any way possible, even
if it's just by jumping into the fire pit.
He will do it.
He will be like, every time I make a step, every time I end
up in a new, in a new state, or every time I make
an action I end up getting a minus two reward.
So what's the point of trying to get to the finish line
if from here will take me two extra steps, I'm just going
to go here and then straight into the fire pit
because that way my reward is going to be less,
the negative reward is not gonna be as bad
as in the case of just making an additional steps.
So you can see that adding this living reward
and depending on the value of the living reward,
that we are adding,
the results are going to be different.
And the agent is going to select different policies.
And that's basically what's, how the reward value, can be,
is incorporated by the bellman equation
even when it's not just at the finish line
or at the end of the game, but even throughout the game.
And again, once again, it doesn't have
to be on every single in every single state
depending on the environment itself, it might be given
to the agent at certain specific states, not at every state
but in our simplistic example, we're just using rewards
at every given state to illustrate this concept.
So I hope you enjoy today's tutorial, and as you can see
we've already made our Bellman equation quite sophisticated
and now it can be applied to many different scenarios
and I can't wait to see you in the next tutorial.
And until then, enjoy AI.