-: Hello and welcome back to the course
on artificial intelligence.
In today's tutorial, we're talking
about the final A in A3C.
We're talking about advantage.
So, there it is.
We've already spoken about actor critic
and asynchronous previously,
and slowly built our way
to what we are going to be looking at today.
And with advantage, we're going to put everything together.
So, this is what we have so far.
We've got a neural network
which is shared between the agents,
the asynchronous agents,
and then we've got the critic
which is also shared between agents.
So, how does this all play out
and why is this critic shared between the agents?
Let's have a look at that.
Well, to understand this better.
We're going to look at an example.
We're going to look at this agent, for instance
and see what happens when he's in a certain state
and he needs to make a decision of what action to play.
So, this agent is in a state, he sees this image,
and then what happens is this information goes
into the neural network.
It goes through the convolutional layer,
then it goes into the pooling layer, then it goes
into the flattening layer, and then from there it goes
into the hidden layers of the neural network.
And then as an output, he gets all of these policy values
the QU values or the policy.
And also he gets the V value, the critic value.
And so, as we know, neural networks, in order to operate
they need to propagate certain errors
or losses back through the network.
So this way, in order to update the weights.
So what weights or so what losses
are we going to be dealing with here?
Well, we've got two losses.
We've got the value loss and the policy loss.
So, the value loss is linked to the V value,
policy loss is linked to policy.
And so value of a loss, we've already dealt with it before.
We know that we have rewards
and we know that we have a discount factor.
So, basically this is very similar to what we were talking
about in the deep Q learning tutorials.
Basically, the network predicts a certain value, V
and at the same time, we can estimate what should be based
on what we know about the environment so far,
we can estimate what should the value V, be in the state
and by comparing the two, we can calculate the value loss
and then back propagate the network to update the weights.
So that's pretty straightforward.
The new thing here is the policy loss.
And so what is this policy loss and how does it work?
Well, this is the part where this whole situation
where the critic is shared between the actors
or between the agents, is going to finally emerge.
So, to understand policy loss
we need to introduce a value called advantage
hence the name of this part of this tutorial,
and this whole part of the A3C algorithm, the advantage.
And the advantage is calculated as Q of SNA minus V of S.
So basically the Q value that you chose to play
of the action that you chose to play
in the state that you were in,
state S minus the value of that state.
So, this is the difference between the two,
and that is called advantage.
And advantage is used
in the calculation of the policy loss.
Now, we won't go into the formula
of the policy loss calculation because it's quite complex.
It uses entropy or it can use entropy.
doesn't have to,
we're not going to dissect that formula
but we're going to understand this on an intuitive level.
Why are we doing this?
Why are we calculating this advantage
and how is it going to help us?
Well, let's look at this for a second.
The Q value here,
comes from what the neural network predicted for this agent.
So, it predicted in this specific action
in this specific state for the actions that it can play.
So, it's got these actions and it can select one
of them and you can play it.
Well, whereas the V value is
or the value that is dictated by the critic
it is the value that we have here in this shared part.
And that's the key here, that this part is shared.
So, the critic, because this is how the critic comes
into play, because we've got a value that we choose
or the action that we choose to play
for this agent in that state.
But then the critic can tell us what is the known value
of that state.
What is overall the known value for this whole group
of agents that are performing together?
Because they're sharing, not necessarily
because they're sharing network,
because they're sharing the critic, they're all contributing
to this, to these V values that are being calculated
for different states.
So, the whole A3C algorithm says,
okay so, the critic knows a V value.
How much better is your Q value that you're
selecting compared to the known V value?
That's what it's saying.
So, that's basically it.
So okay, I'm going to select
a Q value here based on my policy,
based on whatever we use,
like a softmax function
or a epsilon-greedy policy or something like that.
And of course, there'll be exploration
plus exploitation combined in there
but we select a Q value.
And now the question is,
what is the advantage?
Hence it's called advantaage
What is the advantage
that your selected action brings compared
to the known value of that state?
And that is the essence of the advantage.
And basically then that is used to
calculate the policy loss.
And then the policy loss is then backpropagated through
back through the network.
So they're both backpropagated through the network
and the weights are adjusted in order
for the network to better represent the value of the critic.
And also, so that's this top part,
but then also, the key here is that the value
the weights are backpro, when this, the
this policy loss is backpropagated.
The weights are adjusted
in such a way so that this advantage is maximized.
So like that's, that's the intuitive side
of the intuitive understanding of it
that we are backpropagating this policy loss
through the network, in order to help
maximize this advantage.
And what, what that means is basically
that when an agent comes across bad actions
like actions where the Q values less
than the known value for this state.
So basically the whole A3C algorithm knows
that the value for this state is something X
and then all of a sudden you came across a very
bad action and the, and you did a, you chose a bad action.
And what that means
for the A3C algorithm means that, well
why would we do something like that
when it's worse than we already
what we already know about this whole environment
and what we could, could have done
so we shouldn't do more of that, and therefore
the weights are just in a way, so that happens rarer.
So that happens less rare.
So that's a less frequent occurrence that we
choose that bad action.
On the other hand
if you choose a very good action where Q value is greater
than V or much greater than when during this backpropogation
of the policy loss through the network
the weights are gonna be updated
in such a way to reinforce that, to reencourage
reassure that, to happen again
So that the weights will be adjusted in such a way
so the A3C algorithm will think, Oh wow
that that was really cool.
The advantage was very high there, I should do more of that
and therefore it will update the weights
in such a way that will be more likely to occur
in the future, that action.
So, and therefore that is, you know
that's how the network is slowly, slowly going to adapt
and slowly going to construct itself into something that
on one hand calculates the value correctly
and then on the other hand
or as correctly as possible, and on the other hand
it encourages or it has actions which have a high advantage.
So there we go. That's this part.
And now let's have a look at another one just
to kind reinforce what we just discussed.
Let's look at the top one.
So same thing here.
The top agent sees a situation,
a state is in a state and
then needs to decide what to do.
So sends this information to the network.
So this image goes into network,
goes to convolution layer, pooling layer,
flattening layer goes into the hidden layers,
and then from here we get an output.
We get the Q values of the policy, we get the V values.
Again, the same thing.
We've got two losses.
We've got the value loss, which is here,
policy loss which is here.
Value loss we already know how it's calculated
and we discussed this in the deep Q learning
and just discussed this just now as well.
So that's how the value loss calculated.
And then the policy loss, again,
in order to calculate that
which we are not going to go into for now
but on an intuitive level, we're calculating the advantage
which is, okay, so we took a certain action
we chose a certain action based on our selection policy
whether it's softmax or epsilon-greedy
or whatever other selection policy that we're using.
And then what's the action we took?
Now let's compare it to the known value of the state
which comes from the shared critic.
So this critic is kind of like, if you think about it
he's kind of observing all of these agents at the same time.
He's looking at this one, looking at this one, this one
they're all contributing towards the critic to
get the critic more up to speed
with the environment to make sure
that the critic is representative
of what's going on in the actual environment.
So that the weights, that's this is,
that's where the value loss comes in.
So that the weights of the actual neural network
that they reflect very well
the actual situation of things in the environment so
that they can then rely on this value and then use it here.
And so basically, So all of these agents,
All of these agents are contributing
to this critic but then at the same time,
through this value loss but at the same time,
the critic is observing the decisions
or the policies of these agents.
Like, it's like going, looking back at the,
like I'm trying to draw like an arrow to the policy.
An arrow. An arrow.
So looking back at them at the
decisions that they're making.
It's criticizing these decisions through the advantage.
It's saying, Okay, you made a decision.
You chose this, you chose this action.
That's great.
Now let's calculate the advantage.
What does the advantage?
Advantages is A equals the Q value of
the decision I made or the choice I made.
The action I made, chose to take.
Minus the known value to the critic,
the known value to the critic.
So calculate the difference.
If it's a low difference,
your policy then when your policy loss
is backpropagated through the network,
the weights are going to be adjusted,
it's gonna encourage the weights to be adjusted
in such a way that, that doesn't happen again.
That Q value or that Q value is gonna be lower.
So that, because our policy selects the actions
based on the Q values, the higher the Q value
the more likely it's going to be selected.
So if we were using like an argmax policy
then we've just always select
the one with the highest as,
as you remember, we discussed this
then we'd always select the one with the highest Q value.
But we actually, we are using a probabilistic approach
whereas either using like softmax or epsilion-greedy policy.
And then, so we're basically selecting,
we can select any one of them
but the higher the Q value the better.
So if we selected something
and then the advantage was very low,
then bam, the network's gonna be updated
in such a way that, next time
this Q value of that certain action is gonna be less
and maybe something else will be more.
So that's how that, that is played out.
And on the other hand if we select something
where the advantage is gonna be high
then this is gonna go into the policy loss
and then the network's gonna be updated.
So that is a more commonly observed like scenario.
And so basically this whole policy loss helps
the network adapt or morph in such a way
that we do more of the good stuff
good actions and good things,
and do less of the bad things.
And that's how these two losses come into play
and that's how they backpropagate.
So hopefully that clears up in a very intuitive way.
Of course, we didn't go into the formulas
into the complex mathematics behind all of this
and like into the very intricate details
but at the same time hopefully on an intuitive way,
in an intuitive way,
all of this clears up why
we have the actor and the critic
and how they interact together that,
you know, you have these agents asynchronous
so this is the asynchronous side of things.
Then you, this is your actor and your critic
and this is your advantage.
And how that all comes into play.
So these asynchronous agents
they're playing this or exploring the environment
and working through environment
and they're all, altogether contributing to a critic
which is then observing their policies
observing the actors, which is what this is called.
And through the advantage
and therefore coming up with this policy loss
and then policy and value loss
their backpropagated to adjust the network
in order to on one hand represent
the true way of things in the environment.
On the other hand, to improve the actor's performances.
So there we go.
That's a quick recap of the intuition we discussed.
Once again, hopefully this is all
coming together on an intuitive level.
And of course in the practical tutorials
we'll talk more about how all of this works.
I will walk you through this, the process of building this
but having this image in your mind
and this like, kind of like, a roadmap of everything.
How it comes together will be, well should be
I hope it'll be very helpful for you
to better navigate the practical side of things.
And in terms of additional reading for today
we've got two elements.
So first one is on the advantage.
So here we've got
High-Dimensional Continuous Control Using
Generalized Advantage Estimation. by John Schulman.
And this is an image of a stick figure
getting up, like standing up.
And here you can find even more about advantage,
and advantage and you'll find out
the different types of advantages.
So you've got the general advantage estimation
you've got advantages that you use actually
in the formulas in the calculation.
So if you want to find out more about advantage
and exactly how it works, the formulas behind it
and some of the top, top elements or formulas
and know-hows in the space of
this advantage that we discussed
then this is the article to go to.
And one more, one other element that
or piece of work that we wanted to
remind you about is the blog.
A series of blog posts by Arthur Giuliani
which we've mentioned a couple times already.
This is part eight, which is specifically about A3C.
So here you can get
another explanation.
So with a bit more mathematics about what's going on
and you, maybe you can pick up
some additional things from here.
Just two things to keep in mind.
First of all, as always, this blog is TensorFlow
whereas we are using PyTorch.
So keep that in mind.
And the second thing is that the way we structured
our approach is, we talked about actor critic first
then we talked about asynchronous
and then we talked about advantage.
Whereas in his blog, Arthur first talks
about asynchronous and actor critic then advantage.
So keep that in mind.
So hopefully that doesn't throw you off
but other than that of course it's a great piece
of content and we do highly recommend
checking it out for some additional information.
So there we go. Hopefully enjoy today's tutorial
and I look forward to seeing you next time.
Until then, enjoy AI.