In this lecture we are going to discuss reinforcement learning from a more technical standpoint and
this will allow us to define most of the terminology involved in reinforcement learning problems.
I'm a big believer in learning by example.
So in this lecture it's not going to be so much about abstract and technical definitions as it is about
providing examples of everything.
Let's start with the main objects in a reinforcement learning problem.
The agent and the environment.
The best example of this is yourself.
You are in a gym and the world is your environment.
Maybe your long term goal is to ace your math exam and so just like an autonomous vehicle driving to
a destination you must observe your environment and make the correct decisions every day in order to
achieve your goal.
That means studying going to class taking notes doing your homework asking questions when you are confused
and so on.
So that's one basic example
here's another example that's closer to what we might actually use reinforcement learning for.
Suppose you're writing a program that plays tic tac toe.
What's the agent and what's the environment in this case.
The environment is composed of the computer program that implements this tic tac toe game.
Of course this computer program may also involve some form of A.I. that will be the other player in
the game.
But for all intents and purposes just pretend it's a bunch of if statements and predefined rules so
it's not intelligence per say.
It's just a computer program written by someone else just part of the greater tic tac toe program.
You can imagine that this tic tac toe program will have an API that will allow you to interact with
it programmatically.
So for example there might be a function to start a new game.
There might be a function to place your X or your O at some location on the board.
There might be a function to read the State of the board so you can see where all the X's and O's have
been placed so far.
There might be a function to check whether the game is over and if so who won the game.
The computer or your agent.
So that's the environment
your agent on the other hand will be another computer program that interfaces with the tic tac toe game.
Your agent may use an algorithm such as one from reinforcement learning in order to learn how to play
tic tac toe from experience.
It makes use of the API to interface with the environment.
So here we have a program that plays the game.
Part of this code is where the agent reads the state of the game board and chooses the most intelligent
action.
That's our A.I. represented by our agent
here's another popular example video games.
This is a famous classic Atari game known as breakout by the way if you don't know how this game works.
There are many places where you can play this game online for free.
So if you've never played this game before please go and give it a try.
In this game the environment is obviously the game itself.
The goal is to clear all the blocks and your job is to move the paddle in such a way that the ball destroys
the blocks but never falls to the ground.
The agent would be your computer program which can read information from the game like what the screen
currently looks like so it can figure out where the blocks are where the paddle is where the ball is
going and so forth.
Its job is to control where the paddle goes.
So basically you can move left right or do nothing
next.
Let's continue defining more terms.
So far you know about the Asian and the environment.
The next term I want to define is episode what happens when I play a game of tic tac toe or break.
Well some sequence of events will occur and then at the end I will win or lose with my math exam example.
You will take your math exam and then you'll get your grade.
Now we know that with learning algorithms the way that they learn is with data.
So if you're training a dog versus cat classifier you'll need lots of labeled images of dogs and cats
similarly with tic tac toe or breakout.
Once the game is over I can opt to play again.
This is the method through which I will gain experience or to be more technical data.
You might call these games or rounds or matches but in reinforcement learning the official term is episode
so when you're training an agent to play tic tac toe you're going to play multiple episodes and at the
end of each episode your agent will have won or lost.
Hopefully by the end of the training process or in other words after many episodes your agent will be
winning more than losing.
Of course not all reinforcement learning environments are episodic to say an environment is episodic
means that they end at some point and you can start again with a new fresh episode.
Furthermore there is no relationship between one episode and the next so the fact that I lost the previous
tic tac toe episode will have no effect on the environment in the next episode.
However there are examples of non episodic environments.
Take for example the stock market.
For all intents and purposes this can go on forever.
There is no real notion of the end.
Well if you lose all your money then technically there is nothing more you can do.
But it's not the same as losing a game of tic tac toe and starting again.
You can't go back in time and restart the stock market.
Another example is an online advertising system your agent's job will be to choose the right advertisements
to show to users at any given moment in order to maximize your company's revenue.
It should do this continuously.
There is no concept of an end to an online advertising service.
All right so those are some examples of some non episodic environments we can refer to such environments
as having infinite horizons.
All right.
So to recap the terms we've discussed so far now we have Agent environment and episode the next few
terms I would like to think about are the state action and reward.
These items help us describe what goes on when the agent and environment interact let's again use our
tic tac toe example.
In this scenario the state would be the configuration of the board.
So for each position on the board I want to know is there an x there.
Is there an O there or is it empty.
This information is all that I need in order for my agent to make an intelligent decision about what
move to play next.
Speaking of which the moves that the agent makes are what we refer to as the action.
So in tic tac toe to take an action would mean placing a new X or no somewhere on the board finally
the reward is just a number that you can receive at any moment as you play an episode of the game by
the way.
Keep in mind when I say the word game I don't necessarily mean a board game like tic tac toe or a video
game like breakout.
When I say game I mean it in more of a generic sense.
In any case perhaps the reward you get in tech tac toe maybe plus one for winning minus one for losing
in 0 4 draw.
Although this is just an example.
In general you can always assign rewards yourself in order to improve the training of your reinforcement
learning agent.
Here's another example of states actions and rewards.
Think of a maze in this maze.
The state is your position in the maze.
Your actions may consist of the various directions you can go for example up down left or right.
The reward is tricky.
Remember I said it's up to you to think of a good reward to assign to your agent to encourage it to
learn how to solve the environment you might say plus one for solving the maze and zero otherwise.
But ask yourself Is this a good strategy.
Imagine you throw your agent into this maze and it has to learn what to do.
Imagine your agent has played this game ten thousand times and has never solved the maze.
We can pretend the environment is episodic so that after taking one hundred steps you reach a terminal
state and the game is over.
What happens if we get zero reward each time well in that case the agent learns that it does not matter
at all what it does because doing anything always leads to the same reward.
Zero in this case the agent has no incentive to solve the maze.
Your agent will never prioritize one action over another because it knows that no matter what it does
it always gets zero reward.
In this case all actions are equal
perhaps a better reward structure would be to assign a minus one reward at every stage in this case.
You can maximize your reward by solving the maze as fast as possible.
Doing any extraneous actions will lead to a more negative reward not solving the maze will lead to the
most negative reward.
So in this case assigning a negative reward upon reaching any state will allow your agent to solve the
environment in
now you have to keep in mind this is not English class so you have to remove any bias you may have about
what connotations are associated with the term reward.
You might think of reward as a good thing like a prize.
For example if you're a dog and you've just successfully completed a trick your owner may give you a
treat as a reward.
But in reinforcement learning this is not what we mean by reward.
The only constraint is that the reward is a real number.
It can be positive negative or zero you will also receive this number at every step in the environment.
Not just when you reach some goal or definitively fail to achieve it.
The agent as you will learn later in this section will try to maximize its reward over each episode.
For example you may get a reward of minus one hundred.
This is better than a reward of minus one million maybe minus one hundred reward corresponds to successfully
solving the environment but in the end it's just a number.
Don't associate minus one hundred with negative connotation in plus one hundred with a positive connotation
so just remember this.
The reward is not a prize.
The reward is a number which is to be maximized.
You can think of it as kind of the opposite of a lost function.
Whereas we want to minimize the loss in a supervised or unsupervised learning problem in reinforcement
learning.
We want to maximize reward
since I love examples.
Here's one more.
Imagine again the game breakout.
In this case we actually have several options for the state.
For example we may have perfect information about the game.
We could be told the exact positions of all the blocks we could be told the position and velocity of
the ball.
We could be told the location of our paddle and we could be told our current score and the number of
lives we have left.
Although I think you'll find that most reinforcement learning applications do not make use of such information
another way you can read the information about the state of the environment and break out is to look
at the Games ran.
In other words look at the values it has stored in memory.
In contrast to the above this actually is one method used in modern reinforcement learning applications.
It's a proxy to the above perfectly defined state.
You can imagine that it should be quite possible to derive the locations of the blocks and the position
and velocity of the ball and so forth from the values stored in RAM
although I think the most common way to represent the state in contemporary reinforcement learning is
to use screenshots from the game in this way our reinforcement learning a game is learning to interpret
images of the video game just as we do as humans.
I think this is the most meaningful because it's the closest match to how you and I play video games.
We look at the screen you can imagine that models like convolution all known that works would be useful
here
one complication that can arise from only looking at images of the screen is that you don't really have
information about movement an image is only a frozen picture of the game at a single point in time.
Looking at this image how can I tell which direction the ball is moving and so this allows us to consider
an important point the state need not be what I observe in the environment.
It can also be information derived from both current and past observations.
So one way of dealing with the problem of frozen pictures is to simply include past frames as well.
In the famous the U.N. paper they use the most recent four consecutive frames of the game to represent
a single state.
To get back to our states actions and rewards the rest of it is quite basic.
The actions consist of the various moves you can do in the game.
You can think of this in terms of pressing buttons on a joystick or a control pad in breakout.
You can move the paddle left or right for the reward.
As an example you might get plus one reward every time you destroy block
as a final note in this lecture.
I want to introduce you to the concept of state spaces and action spaces.
This is important as we move down from high level ideas and concepts to the actual math that will allow
us to solve reinforcement learning problems the particular math concepts that we need to describe state
spaces and action spaces is the set the state space is the set of all possible states and the action
space is the set of all possible actions.
We don't need to go further than this.
We just need to know what it means.
So as an example consider the canonical example of a reinforcement learning problem known as grid a
world in grid worlds.
The idea is you're going to start in the bottom left square and your goal is to arrive at the top right
square with a ruby is if you make it there you get a reward of plus 1 below that there is a losing state
where if you arrive there you'll get a reward of minus 1 and in the second row second column there is
a wall meaning that your agent cannot go to that square so that's the basics of the game to describe
the state space.
That's simply the set of all possible positions on the board.
So you may want to pause this video and look closely at this list of coordinates to confirm that they
correspond to positions on the board the action space consists of the actions up down left and right.
Now the reason we had to talk about grid world for a bit is because for our other examples such as tic
tac toe and breakout the state space is much more complicated.
The action spaces are pretty simple since for tic tac toe it consists of all the possible positions
you can draw an X or no.
And for breakout it consists of moving left right or doing nothing but for tic tac toe the state space
is quite large as there are many possible configurations of the board as an exercise.
I would strongly recommend trying to write a computer program that can enumerate all the possible configurations
of a tic tac toe board.
This should give you some intuition about why games like chess and go are very difficult.
You can imagine that if a three by three board with only two possible characters can have thousands
of states imagine how many states are involved in chess and go
for breakout.
The number of states is even larger.
It's equal to the screen resolution multiplied by the number of possible colors per pixel two to the
power twenty four or two to the power eight to the power of three.
But for all intents and purposes we can consider images just like Time series to be continuous valued.
The only reason they appear to be the screen is because computers have finite position and so the values
need to be quantized.
When we have continuous values this means that the number of possible values is actually infinite.
In fact it's possible for actions to be continuous also so that the action space is also infinite
since this lecture was quite long.
Let's summarize everything we learned.
This lecture was all about defining some reinforcement learning terminology to help us in our discussion
of reinforcement learning.
First we define the terms agent and environment.
You can think of the environment as the world or whatever computer game you are teaching your agent
to win.
You can think of your age as your computer program the one that does the learning.
Next we define the term episode.
This is like one round or one match of a game as you know machine learning models learn through data
or in reinforcement learning parlance experience and so you can imagine that in order to sufficiently
learn how to play a game.
This is going to require multiple episodes.
Next we learn about states actions and rewards.
Rewards are a number which can be any number positive or negative.
The job of a reinforcement learning a gym is to maximize its reward actions are what an agent does in
an environment.
For example playing a move in tic tac toe are going left or right.
In a video game states are what we observe from the environment but they can also be values derived
from those observations or even a sequence of past observations.
To add a little more to this we call the last State of an episode a terminal state so when you reach
a terminal state that is the end of your episode.
Lastly we define the terms state space and action space.
These are the set of all states and the set of all actions respectively using these terms.
We can now talk about reinforcement learning coherently and build a framework that allows us to solve
reinforcement learning problems.