In this lecture we are going to discuss the concept of states actions and rewards.
More in-depth this lecture is about how we encode states and actions when we are programming talking
about states actions and rewards theoretically is nice but eventually we're going to have to put this
into code as a side note.
Since rewards are just numbers this is trivial.
There is no need to discuss how we represent numbers in code
let's start with the state.
As mentioned previously the state can be discrete or continuous in a game like tic tac toe.
The states are discrete because they are just different configurations of the tic tac toe board.
If we make a robot that has sensors like a camera a microphone a gyroscope a proximity sensor and so
forth.
Those would all be continuous values so you would end up with a vector of continuous values.
In fact this takes us back to regular supervised learning.
If our targets are categorical how do we represent them in code.
If there are key categories we will use the integers 0 up to K minus 1.
So we might say that dog equals zero cat equals one mouse equals two and so forth.
Obviously it does not matter which category is assigned to which number.
And the reason we want to do this is because at some point we're going to have to use these categories
as Array indices.
And so similarly if we have big states then we'll represent them in code using the integers 0 up to
a big S minus 1 for continuous values.
It makes sense to store them in a vector although if you have something like an image then it will be
a three dimensional tensor.
So more generically if you have continuous values you can think of them as a tensor with one or more
dimensions
now that you know how to represent states and actions in code.
It's time to talk about policies.
When I first started learning about reinforcement learning I found policies to be kind of an odd concept.
Well actually I found all of reinforcement learning to be kind of odd and foreign but you'll get used
to it.
The idea of a policy makes sense at a high level but it becomes somewhat ambiguous when you start thinking
about how to represent it mathematically or in code.
The policy is what the agent uses to determine what action to perform given a state.
It's important to keep in mind that the policy yields and action using only the current state it doesn't
use any combination of the current state and previous states and it doesn't use any information about
rewards technically as I mentioned earlier.
The state may be made up of multiple observations and that may also include the reward although that
is unconventional but strictly speaking the policy will yield an action using only the current state
the simplest way to think about a policy is that it's a dictionary mapping or a function that returns
in action.
Given a state here is such a function as you can see the only input is a state s and it returns an action
a where the state s was the key to the dictionary and the action a was the value the real question is
how can we represent this mathematically
this is why it's useful to talk about how we encode states and actions first.
So imagine again we are in grid world your agent now has a dictionary representing what action to perform
given the state as you can see here.
So for example if we're to the left of the goal state then the appropriate action is to move right and
just to be clear the left of the goal state is 0 2 if we're at the initial state then the appropriate
action is to move up and just to be clear that is the state 2 0 now from the initial state.
Moving right is just as valid since we can still reach the goal from that point.
You can see that although I've encoded the states here explicitly as tuples we could gain more efficiency
by encoding them as integers corresponding to the tuples and using those integers to index an array.
As you may recall from your computer science studies indexing arrays is faster than indexing dictionaries
thinking about policies as dictionary mappings is somewhat limited.
There are two reasons for this.
The first is that this won't work.
If you have an infinite state space you would need an infinite sized dictionary which is not possible
to have.
Second is that it doesn't allow our agent to explore its environment.
Think of training your agent like teaching a baby at first.
A baby knows nothing.
It must try new things in order to figure out how the world works and build up its intuition a reinforcement
learning agent is the same way.
If it has a fixed policy and it only does the same thing all the time then it can't gain new experiences.
Thus it makes sense for policies to be stochastic stochastic is just a fancy word for random.
In other words a more general way to represent policies is to represent them as probabilities
representing policies as probabilities actually solves both of the problems I posed above.
Let's see how first it deals with this problem of randomness.
The common way of dealing with this in reinforcement learning to allow the agent to explore is to give
it a small chance of performing a random action.
So here is a python function that can accomplish this.
We first generate a random number.
If this number is less than some small number Epsilon let's say zero point one then it will choose an
action at random from the action space otherwise we will grab an action from our fixed policy dictionary
mapping.
This method is called Epsilon greedy and you'll learn later in this section of why it's useful and what
the relevance of exploration is.
So what about continuous state spaces.
In fact seeing policies as probabilistic easily lends itself to continuous or infinite state spaces.
Imagine your state is a vector s.
Now imagine we have some policy parameters w the shape of W is the dimensionality of the state space
by the size of the action space.
For now we'll presume that the action space is still categorical So what do we do when we want to output
probabilities for a set of categories.
In fact this is just like classification we can use the soft max function.
So now our policy is the soft max of W dotted with the state s as you can see.
This allows us to introduce a little more notation.
It's common in reinforcement learning to denote the policy with the symbol pi not to be confused with
the number pi.
For any given state we can calculate a probability distribution over the actions base Pi of a given
S. then to decide which action to perform in the environment.
We can simply sample from this distribution.
This allows us to explore if necessary but we can still treat this policy deterministic Lee.
If we want simply by using the ARG Max also note that it's not necessary to use a linear model as we
are doing here.
We can't in fact use any function approximating such as a neuron that we're
now at this point.
You may be wondering how can an intelligent agent possibly know what to do using only the current state.
This is the problem we described before just by looking at a still image of the road how can I know
what the correct action is if you're more inclined to think inside a supervised learning paradigm.
You may insist that there should be a target here so that the agent learns to associate this state with
this action but in fact it's quite possible for an agent to learn how to plan for the future using experience
collected by playing multiple episodes even without an explicit target for a given state.
The agent can still learn what action to perform such that it maximizes rewards in the future.
This is what we will learn about in the coming lectures.