-: Hello and welcome to this tutorial.
Okay, so we just computed the entropy
and added it to the entropies list,
and now what we're gonna do
is take a random draw of an action
according to the distribution of probabilities
of he SUFMACS.
So let's do this.
That's the next step.
We are still in the loop
because we're still running on the steps here.
And so you now know how to play the action.
We will first introduce a variable
for the action, called action
and then we take our distribution of probabilities
and we're gonna use the multinomial function
to take a random draw from this distribution
of probabilities.
And then we add that data.
So it's important to note
that the action will actually be a tensor
with only one value
but you should not see this as a simple value.
You should see this as a tensor of dimensions
one by one that contains this value for the action
and that's because it is unsqueezed.
And now, still in the same for loop,
we are gonna get the log probability
associated to the action that was displayed.
And so I'm updating my log probability here
by taking the previous one
the previous log probe that we computed here.
And then I'm gonna use the gather method
to which I'm going to input one
and the action that was displayed, because we want to
get the lock probability that is associated to this action.
And so as a second argument here, I'm gonna input my action
but that has to be as a torch variable,
as required by the gather function
and the gather function just indexes with a tenser integer.
All right, so now we just got the log probe associated
to the action that was displayed
and now the next step is to append what we got
to the list here.
So we got the value
that's what we got here as the output of the model.
Then we also got the log probe.
So we are gonna add the log probes to the log probes list.
We already appended the entropy to the entropies list.
So, we're good.
And the rewards, we will get it afterwards.
So we will now append the value and the log probe
to the values list and the log probes list.
Let's do this.
We take our values list, we add dot,
we use the append function and we add the value
that was just returned by the model.
Perfect. Then same for the log probes.
We just got our new log probes,
and we are going to append it to the log probes list.
And so in this append function, we input log probe.
Our log probe, that was just computed here.
All right, so our lists are now well updated.
So now what we're gonna do is play the action
because actually right here we selected the action,
by taking a random draw from this distribution
of probabilities here,
but we actually haven't played it yet.
And we are gonna play it now
so that we can reach the new state
and therefore get the new transition.
And to play it, we're gonna take our environment
because we play the action in our environment.
Then we're gonna use the step method.
And inside we specify the action
that was selected to play it.
And to do this, we take our action
and we add .numpy
because that's what is expected by the step function. Okay?
But this returns
actually the new state
and also the new reward because by reaching a new state
we get a new reward,
and also we get a new value
for done to know if the game is done or not.
All right?
So with this, we play the action
we reach the new state and we get the new reward
and we know if we're done with the game.
And speaking of being done with the game,
well, we're just gonna add something here
that will make sure that an agent is not stucked
in some state.
And to do that
we're gonna update the done variable the following way.
Well, it's gonna be equal to done
or we're gonna add a condition
saying that the episode of the game
should not last too much time.
And it will see in the main function
that there will be a max episode length parameter
which will be equal to 10,000.
And we don't want an episode to last more than 10,000 units.
So we're gonna add here episode,
length,
which is the length of an episode,
and we're gonna write the condition
larger,
than max,
episode length.
There we go and actually
max length, we are getting it from our parameters,
therefore I'm adding here perams. Perams.maxespisodelength.
So this means that if the game is done,
or the length of the episode is larger
than the maximum length of episode set,
which will be equal to 10,000, well the game will be done
and we will start a new game.
Okay? So that's just a precaution.
And speaking of precaution, we're gonna add
another precaution.
It's to clamp the reward between minus one and plus one.
We already got the reward here,
but we want to make sure that the reward
is between minus one and plus one.
And to do this, we simply need to update the reward
by doing this: taking the max,
then taking the min of reward
and one.
And here we take the max of the minimum of reward and one
and minus one.
And that will make sure the reward is between minus one
and plus one. All right?
So another precaution,
and now we just want to check if the game is done
in which case we will restart the environment.
And why do we need to check that now?
It's because we just reached a new state.
We just passed a new transition.
So we need to check that after passing this new transition
while the game is not done.
So if done, again- If done, then in that case,
we will restore the environment
by setting the episode
length,
to zero.
And also the state will be re initialized.
And to re initialize it,
we take our environment and we use the reset function.
Okay, now we get out of this if condition.
That was just a checking.
And now what we're gonna do is since we reached a new state
while this new state is right now, MPIArray
because remember, the states are the input images
which originally are MPIArrays.
And so now what we have to do is to convert the new state
into a torched tensor.
So we are going to update our state
and we're gonna use the torch library,
and of course the from numpy
function
to convert this numpy erased date the input images
into a torch sensor.
Perfect.
And now the last thing we need to do
before getting out of this for loop,
that is the loop on our steps.
Well, it's to of course
append the reward to the rewards list.
That's the last thing that needs to be updated.
We updated, all the lists here except for the reward.
So we're gonna do that right now.
We take our rewards and we use the append function
to append the last reward that was just received.
Perfect.
And just before we get out of the for loop
we just need to do one last check to check that.
If it's done, then we want to stop the exploration.
And so we're simply going to add here a break.
Meaning that if it's done, we stop the exploration
and we directly move on to the next step,
which will be the update of the shared model.
And now we are done with this for loop.
Now that the agent has done its exploration,
it'll update the shared model.
And we will take care of that in the next tutorial.
Until then, enjoy A.I.