Earlier in this section I told you about the value of treating the policy as a probability distribution.
First it allows us to describe the entire MBP as a set of two probabilities one representing the environment
dynamics and one representing the agent.
This allows us to reason about the MVP mathematically and find a solution to both the prediction and
control problems as we just did.
The other important reason which I mentioned briefly earlier is that in order to find the best action
we must first know the result of performing different actions.
In other words we must have collected those samples.
Here is the problem when we take the ARG Max overall actions queue of essay The Queue value here is
not changing that much from one episode to the next.
Let's say we have three actions so we want to choose from queue of as a one queue of essay 2 and queue
of as a 3.
While perhaps we initialize queue of as a 1 and queue of as a 2 to 0 while we initialize queue of as
a 3 to 1 also assume that all the rewards are positive during the process of playing the game.
Maybe a queue of essay 3 gets updated to a value of 2.
We will now always choose action 3 because that gives us the best queue value out of the three possible
actions.
The problem is we don't know the true values of queue of essay 1 and 2 of essay 2 because we never collected
data from performing those actions.
How do we know the values of the other actions.
If we have never actually performed those actions.
In fact we don't know those values at all.
We just happen to initialize them to zero because of that.
There is no way for us to confidently choose the right action.
This is a more general problem known as the explore exploit dilemma.
Suppose you go to a casino and you're playing a simplified version of slot machines.
You have several slot machines to choose from and you don't know which one you should play.
However you do know that not all these slot machines yield the same reward.
As I said these slot machines are simplified so there are only two possible outcomes.
You can either win or you lose if you win you get one dollar and if you lose you get zero dollars.
Obviously you would like to play the slot machine in which your chances of winning are highest.
OK.
So how can you figure out which slot machine will give you the best chance of winning.
Well you cannot ask the manager of the casino because he wants to make money.
He will not share that information with you.
So what do you do.
Your only method of figuring out which slot machine is best is to collect some data.
So let's say you play each slot machine 1000 times the win rate of the slot machine is simply equal
to the number of times you won divided by the total number of times you played
but what's the problem with what we just did let's say you have to choose between five slot machines
if you played a slot machine 1000 times that means you've played 5000 games.
Now let me remind you that playing a slot machine at a casino is not free.
In general collecting data is not free.
It usually requires time resources or both.
Let's say each time you play a slot machine you spend 25 cents.
You need to win at least one out of every four times in order to break even.
Imagine that for some of these slot machines.
You win zero times.
Now it becomes clear that playing those slot machines 1000 times would be a huge waste of both time
and money
here's another question to consider.
Why should we collect 1000 samples per slot machine.
Why not one hundred Why not ten.
As you know the more samples you collect the better and more accurate your estimate becomes unfortunately.
This also equates to more time and money spent.
That's why we call this a dilemma.
The explore exploit dilemma means we are trying to balance both exploration and exploitation.
We want to explore so that we can collect more data to find out which slot machine is truly the best.
On the other hand we want to exploit because we want to play what we believe is the best slot machine
in order to win more often and to make more money.
These two forces always oppose each other
the typical answer to this problem in reinforcement learning is called Epsilon greedy.
This allows us to be greedy and exploit what we think will yield the highest future sum of rewards.
But we have a small probability epsilon of choosing a random value so that we can explore and collect
samples for our Q table to make our estimates of Q more accurate.
And remember they do need to be somewhat accurate in order for us to confidently choose the best action
here's how we will do this in code instead of always performing the action specified by the policy we
will generate a random number between 0 and 1.
If this random number is less than epsilon we will choose an action at random.
Otherwise we will perform the action specified by the greedy policy which is greedy with respect to
Q Given the state s.