Instructor: Hello and welcome to this tutorial.
Congratulations again for being done with the A3C.
We made it, we made the brains and we trained them
but now we still have to make a test agent
which will not update the model at all,
but it will just use the shared model
to make its own explorations.
And of course, in this code we will record some videos
and these will be the videos
of this test agent playing Breakout with a certain score.
So let's go through this code.
The most important is done so,
as I told you, we're not going to code it line by line
but I think it's important
that you understand what's going on here.
So there we go with this code.
In the first section, as you noticed
we import the libraries
and then we define this test function
which will make this test agent do its own exploration
and play the Breakout game.
So there we go.
This test function takes three arguments.
The first one is rank,
and that is still to de synchronize the test agent
as we did for the training agents.
Then we have our parameters of course
because we will need some.
And of course we have the shared model
because this test agent will use the shared model
to do its own exploration.
All right, then there we go inside the function
in this line of code
we de synchronize the test agent
exactly as we did before.
Then we import the environment.
So I remind that in the main code
which will be in the next tutorial,
we'll end name here will be replaced by
Breakout V zero,
so that we can go into the Breakout V zero environment
and play the game.
And video equals true means
that we will get the videos of our AI playing Breakout.
So basically this line of code in total
means that we run one environment with a video.
Then at the next line of code
we de synchronize this environment.
So exact same principle as in a train function.
Then we get our model.
And to do this, we create an object of the act credit class
and we input the input shape
with our environment, observation space, and shape zero.
So exactly like for the train function.
And our outputs which are the actions with end action space.
So exactly like before, then something new here
since we are done with the training
we don't want to put the model in train mode
because simply we don't want it to train.
We want to put it in eval mode.
And that's what we do here with model.eval.
So that's just basically to put the test agent
in a mode that will basically test it
test its performance, evaluate it.
Then here we get our input states
which are the input images from the game
which at this point are none by race.
Then here we convert them into torch dancers.
Here we initialize the sum of the rewards
here we initialize done to true.
So still just like last time, then something new again.
We introduce this third time variable
with time to time function
to measure the time of computations.
And that's because we want to get the starting point.
Then here for the actions, we use a very practical type
of queue that allows to add an element to the queue
from the right or from the left.
So that's very practical
and I'll give you the reference link
in the commented version of the code.
So you'll have a look at what this dequeue is
and that's what allows to do that.
Then we initialize the lengths of an episode
with zero of course,
and then we will increment the size in this well loop.
So we use the sim trick here.
While true and in the loop we increment the length
of the episode by one.
When the game is done, when the game is over,
we reload the last date of the shared model,
the shared model that was updated by the other models.
Remember that here, the shared model is no longer updated.
Then still if the game is over, if the game is done,
we re-in it, we re-initialize the cell states, CX
and the hidden states, HX.
And else if the game is not over,
well we keep the same cell states and hidden states
but we make sure they are in a torch variable
so that they can be attached to a gradient.
Okay, so that's something we already did
in the train function and then still in the well loop.
And after having updated the cell states
and the hidden states the right way,
depending on the two cases here, well what do we do?
We get the predictions of the model.
So that's exactly what we do here with this line of code.
So we get the value
which is the output of the critic,
the action value which is the output of the actor,
and then the top all of the hidden states, HX
and the cell states CX.
Then we generate a distribution of probabilities
of the actions that is of the Q values, action value here.
And we do this with the soft max function.
And of course we don't need to get
the log probabilities here,
because this is just for the training for the test agent.
It will just play the actions.
Which will just use, you know,
like Doom a soft max body to play it.
But we are not doing any training here.
So we have just the prob.
And from this prob, we play the action
by taking directly the arc max of these probabilities.
That is it takes the action
that has the highest probability.
And the reason is that
the test agent doesn't do any exploration.
Remember that we want to give a chance to pick
up some actions that have low probabilities
when we want to do some exploration of these other actions
and you know, not taking each time the action
that has the highest probability.
But here the test agent won't do any exploration
and therefore that's why we directly
take the action that has the maximum probability.
Okay, and then once we play the action
we reach the next state and we get the next reward.
And done is updated whether or not the game is over.
So this, we get all this
with this line of code
by playing the action after
having selected it with the arc max here.
So we play the action here and we get the state
we get the reward and done is updated.
Okay, and then since we just got a new reward
we're gonna update the sum of the reward
by simply adding this new reward
and finally whenever the game is done,
so if done means when the game is done
when the AI finishes playing the game,
well we're gonna print the results
with the time,
the episode reward,
the lengths of the episode,
that is how much time did it last playing Breakout.
And this is just how we print all these variables
Using these time tricks.
So that's for the time.
Then reward sum is just a variable
for the sum of the rewards.
And episode length is the variable
for the length of an episode.
Okay, and then once we printed all the results,
well since the game is over and we wanna start a new game
we're gonna re-initialize everything that is,
the sum of the reward to zero,
the length of an episode to zero.
We are gonna re-init all the actions
by using this clear function,
reset the input images you know,
by re-putting all the breaks all together.
And finally, we use this time, that's sleep,
60 seconds to just do a break
of one minute to let the other agents practice.
And that's if the game is over.
Okay, and finally, we have this last line of code,
which will get us the new state
and then we can move forward.
We can continue in this new game.
So there we go.
That's the test function,
thanks to which we will see the videos
in one or two tutorials.
I hope we will be all together like last time
to watch the results,
that is with you, Carol, and me.
That will be fun.
And I'm telling you, expect to see good results.
But keep in mind,
that this Breakout game was super challenging.
We thought it was the simplest game to play first,
but not at all.
It actually turned out to be much more difficult than Doom
and that's why we put it in the last module.
But anyway, let's make this main function
in the next tutorial.
Saying, this is not the most important here now that
the A3C is all implemented.
So we will not code it line by line
and we'll expand the code
and very quickly we will get to the results.
Until then, enjoy AI.