In this lecture we are going to define a one more term which deserves its own lecture so far.
I told you that the goal of the aging is to maximize the reward it gets.
But as you recall the reward may be structured differently in different games.
For example in tic tac toe you may receive a plus one for winning or a minus one for losing on the other
hand.
If you're solving a maze you might get a minus one at every step.
So what does it mean to maximize the reward.
Does it mean to maximize the reward at the next step.
Or does it mean to maximize the total reward over an entire episode.
Here's the answer to stated more accurately the goal of the game is to maximize the sum of future rewards.
Why is that.
Well it can't maximize the rewards it got already.
Those are in the past.
They cannot be changed.
Furthermore we don't want to only maximize the reward on the next step.
What if we are solving a maze and we get a minus one for any step we take.
In that case the agent isn't incentivized to do anything useful because no matter what it does the immediate
reward is still minus one.
Thus the agents true goal is to maximize the sum of future rewards until the episode is over.
In this way the agent is planning its future steps as well.
It must have some concept of where it will end up because that's the only way it will know how to maximize
rewards in the future
to take a real world example.
Consider again the idea of preparing for a math exam for a math exam.
You don't receive any reward until you've completed the exam.
The reward signal is your grade on the exam.
But imagine all the actions it will take to actually maximize that reward.
You'll have to study you'll have to do homework.
You have to forgo socializing with your friends.
In fact all of those actions do not sound very rewarding at all and thus if your only incentive is immediate
gratification.
In other words the rewards you receive immediately.
Then you will not do well on your math exam.
Instead you must make use of long term planning.
Sure I may not want to study today.
It may be very annoying and I will miss my favorite TV show.
But because you are planning long term you're not thinking only about today.
You're thinking about the results of your math exam.
The desire to maximize the total future award is necessary for long term planning
we call the sum of future rewards the return we describe the return mathematically.
Using the symbol je because it depends on future rewards only it is time dependent.
So we index it with a T.
We can say the return at time t is the sum of rewards at time T plus one up to the terminal state a
time a big T
now you might be wondering what happens if we have an infinite horizon MVP a game that never ends.
In this case your return might be infinity.
Therefore we introduce a concept known as discounting discounting is used for infinitely long tasks
but it's also used for episodic tasks as well.
We introduce a discount factor called gamma.
Each feature reward is weighted by gamma to some power gamma is usually a number close to one like zero
point nine zero point nine nine or zero point nine at nine.
It's a hyper parameter so you'll have to choose its value based on the performance of your agent.
The idea is the further you go into the future the harder it is to predict.
Therefore we care a little more about getting rewards now than we do later.
Intuitively this works just like money.
I would rather receive one hundred dollars today than receive one hundred dollars ten years from now.
In ten years 100 dollars will be worth much less than it is today due to interest
one important feature of the return which we will make use of throughout the rest of the section is
that it can be defined recursively.
In other words in terms of itself specifically the return at time t is equal to the reward at time T
plus 1 plus gamma times the return at time T plus 1.
This may not seem like much more than a simple math substitution now but you'll see how it will become
very useful later on.