-: Hello and welcome to the third module of this course,
The A3C,
Asynchronous actor critic agent.
And so now I can really say welcome,
To the state of the art, machine learning.
Well, at the time I'm saying this because maybe some,
Of you will take the course in one or two years,
But at the time, I'm saying this in 2017.
While you are about to work on one,
Of the most powerful model in artificial intelligence,
But there is more,
That is not the only special thing about this module.
Not only we're about to work
With the most powerful AI model.
But we are going to implement the most powerful version,
Of this algorithm, that is the most optimized version,
Implemented version of the A3C model.
Because you can imagine that there is the heart,
Of the A3C algorithm, but then there are a lot,
Of tools that we can use to optimize the whole model.
And so not only you're gonna have the heart,
Of the A3C algorithm,
But who are also going to implement all these tools
Around it to make the model super powerful.
And why did I want to do that?
Well, that's for two reasons.
The first reason is,
That we are approaching the end of this course.
This course is the highest level course,
Between the three courses, Ml, DL, and AI at a Z.
So now I think you're ready to take it at the next level.
And the second reason is that,
Solving breakout is actually super challenging.
Remember in the promo video,
We wanted to put breakout as the first module,
Because we thought it would be the easiest challenge,
But not at all.
It was actually the most difficult challenge.
An easy way of explaining this, is that, well in Doom,
The monsters are big and therefore easier to detect.
And therefore easier to kill or avoid.
But in Breakout,
We have this tiny ball that the AI has to detect as well,
Because the AI will still have eyes here.
We're still going to do deep reinforcement learning.
So it's actually super challenging
And that's why we don't really have a choice,
To implement the most powerful version of the A3C.
Now, why do I say this is the most powerful version?
That's for a particular reason.
It's not like I'm saying that, I'm going to,
Implement the most powerful version of the A3C.
Now, it's not this, The reason I'm saying this is that,
The version we're about to implement,
And this is something very special we're gonna do,
Is actually a version of the A3C that was implemented,
By somebody, but corrected by,
One of the most influential people,
In machine learning today,
Who happens to be the creator of PyTorch.
His name is Adam Paszke.
So now what we're gonna do
We're gonna go on GitHub on the PyTorch main page.
And if you scroll down down to the end
Down to the bottom, you will see the team,
The team of Pytorch creators and contributors.
And you can see here that PyTorch is currently maintained,
By Adam Paszke.
That's the person who we should really be grateful for,
Because there are very few versions, of the A3C,
That work well for Breakout.
And he corrected one of the code,
For the A3C to make Breakout work perfectly well.
So Adam Paszke is not only maintaining PyTorch,
But also he's one of the creative PyTorch.
And as I said today, he's,
In the top 10 most influential people in machine learning.
So we can feel confidence,
That the version we are about to implement,
Is probably the most powerful version of the A3C today.
And so what is this implementation?
Well, originally it comes from,
A developer called Elia Kostrikov.
And so as you can see,
He did a PyTorch implementation of the A3C,
Which originally didn't work well for Breakout,
But then somebody made the pool request.
If we go to the pool request here,
We can see in the close one that, there we go.
We have this one, a cleaner solution to G sharing problem.
And guess who this pool request was made from?
It was made from Adam Paszke, the Creative PyTorch.
And that solved the problem, that's made the A3C,
Work very well on Breakout,
Without waiting for days and days.
And therefore if we go back to this implementation,
We can see the four contributors,
Of this most powerful implementation.
And here are the contributors.
So thank you very much to all of them,
And we can say a huge and special thank you,
To Adam Paszke for fixing the gradient sharing problem.
He started by doing a fork, which is a sub branch,
Of the code, and then he did a pool request,
To the developer to fix this problem.
There was in the code, which is a gradient sharing problem,
And that's how he became a major contributor,
Of this implementation.
Making the whole thing work perfectly well.
And trust me, I did a lot,
Of experimentation on the A3C model.
I actually implemented five models.
I was even desperate that it didn't work well.
So I made my own Breakout on KIV to have a bigger ball,
And therefore an easier pre-processing of the images.
Then I went back to open AI and made my own implementation,
Of the A3C, but that took ages, to run,
And train on a pretty powerful computer.
So I wanted to find a better way.
And that's the way it is, this very powerful implementation,
Of the A3C model of which one,
Of the major contributors is the creator of PyTorch.
So what we're gonna do in this module,
I think you're ready for that.
Is implement this highest level code,
For the implementation of the A3C.
So we're basically going to re-implement all these files,
And we will mostly insist,
On the files that are directly related to the A3C.
For all the parts that are directly related to the A3C.
We will implement the code line by line,
For the others I will just explain the code.
So we should be able to tackle this,
Without finding it too overwhelming.
So there we go.
Quite a special module.
Not only we work on a state of the arts model of AI,
But also at the time I'm speaking,
I'm highly confident we're implementing the most,
Powerful version of the A3C.
So let's do it.
Let's go back to Python.
And let's start all this,
Before we start.
We're gonna do the most simple thing we're gonna do
In this module, setting the working directory folder.
So let's go to our AI a to z template folder.
Module three Breakout, the most challenging one.
And there we go.
That's all our files.
So let's see which ones are directly related to A3C.
And so let's see which ones we're going to implement,
Line by line and focus our energy on.
So there are actually two fouls.
The first one is Model .py, which is this one.
So we will re implement it line by line,
Because that's the most important.
That's where we make the A3C brains.
And the most important thing to understand here is,
That we will have a shared model which will have,
The same update of the weight for the actor and the critic.
So that's a part of the special version of the A3C,
The shared model with the shared update of the weights.
And then the other most important foul,
That we will implement line by line,
Is the train.py file, of course,
Right after we made the brains of the A3C,
Well we have to train them,
And we trained them in this train.py file.
So this is quite a long code,
But this is what contains the heart of the A3C model,
Which will have two losses to reduce the value loss,
Which is the loss related,
To the predictions of the critic and the policy loss,
Which is the loss related to the predictions of the actor.
So this is quite new,
But you know, that's because in the A3C,
We're basically working with several agents,
Each one having their own copy of the environment,
But we also have this fully connected layer,
That outputs a value of the V function,
And that basically is a common vision,
Of what's happening in the game.
So, this will be quite challenging.
So make sure to be in good shape.
And for the rest of the fouls,
well I will just explain them in details
But not by spending too much time on them.
Believe me, you want to keep your energy for this.
This will be already a lot.
So these fouls are,
Ends .py.
Which is an improvement,
Of the gym environment thanks to Universe.
So basically,
That foul just improves the gym environment with Universe,
And that allows to have an optimal pre-processing,
Of the images,
And also to normalize all the values of the environment,
Like the colors intensities or the rewards intensities,
While all the values of the environment.
This file normalizes all these values.
And also make sure we have an optimal pre-processing,
Of the images.
And as you can see, this is taken,
From this open AI GitHub page with a Universe star agent.
So we will not spend too much time on this.
We will actually stop here.
You just need to understand,
That we improve the gym environment with Universe,
To get an optimal pre-processing, of the images.
The rest is not that important, especially for the A3C.
Then we have main.py
Which is the code that will execute the whole thing.
So you know the code that will run the whole thing,
Create the brain, train the brain and output the videos.
And that's because it will run all these codes here.
So model, we saw what it was then myoptin.py,
Is a special optimizer that's basically the atom optimizer,
But adapted to this shared model that we're implementing.
So we will explain all this code in one tutorial.
Then we have test.py, that's actually the last one.
So test.py is basically the file, that will,
Implement a test agent.
So that is an agent, that will play Breakout,
Without updating the model.
So that's totally independent from the training.
And we will also explain this code in details.
Besides the good news is that you will have two codes,
One code, which will be the code we implement,
In the tutorials, but without any comment.
And one of the code that is one,
Of the code folder with all the codes commented.
So with all these six valves, all well commented,
So that if you miss something in any tutorial,
Well you will be able to look,
At the commanded code to understand what's going on.
So there we go.
I hope you're excited to implement this.
You're really at the top of the mountain now or just<
Below the top because you need to understand this first
But you're getting there to take a good breath of oxygen.
And there we go for this super exciting journey.
Until then enjoy AI.