-: Hello and welcome back
to the course on artificial intelligence.
Today we're talking about the first part
of A3C, the actor-critic part.
So, here we've got asynchronous advantage
actor-critic algorithm,
and we're going to be talking
about that underlined actor-critic part.
That's where we're going, we're going to start.
You could technically start anywhere,
but it just makes a lot more sense to start
from actor-critic, because that way
we'll have like a very consecutive explanation
of or intuitive understanding of what's going on.
It's going to facilitize better
if we start surprisingly at the at the end
of the separation.
All right, so, so far in this course we've come
up with Deep Convolutional Q learning
which is illustrated over here.
So we've got the computer seeing the pixels
so the actual image and pixels, not just the vector.
So it's not cheating.
It's actually seeing exactly what a human sees.
It sees the monsters, it sees the health,
it sees the parameters at the bottom, it sees the corridor
it sees the gun, it sees exactly the same thing
as a human would see when playing this game.
Then that image is passed through a convolutional layer.
Then it's passed through a pulling layer.
Then it's flattened and goes into a neural network.
And then at at output, we've got actions, as you remember
we've got those Q values.
Then we apply a action selection policy to them.
So for instance, we apply a soft max
and we find out which action we want to take.
And so there's some exploration
plus exploitation going on there, a combination of the two.
So that is how Deep Convolutional Q learning works.
But now let's see what we're going to do with this.
So for simplicity sake, just so that it's easier
for us to operate with this, because we're going
to adjusting this image and moving it around
we're going to replace the circles with squares, with these
with these rectangular boxes.
And we're also going to get rid
of those lines in between, just changing them to arrows.
So this doesn't change the essence.
This is just the representation on this chart.
This is still, even this representation is still Deep
Convolutional Q learning is just going to be easier
for us to modify it and show exactly what A3C is.
So that's just how we're going to represent things
from here and what A3C does or this specific part,
so we're starting, remember
we're starting like step by step.
We're starting with the actor-critic part.
So we're going to see how we go
from Deep Convolutional Q learning to A3C step by step.
And first step, we're going to introduce this,
this actor critic part over here.
So we're going to talk about that.
So the first thing that happens is this last bit
the output is actually,
we're just going to redraw it like this.
So it's exactly the same output,
exactly the same Q values or exactly the same actions.
So if he, if you had eight possible actions
you still have eight possible actions.
We're just gonna put them
at the top so they take up less space.
So nothing, so far, nothing has changed.
So, so far this, and this are exactly the same.
But now this is where the actor-critic part comes in.
We're going to have a second output.
We're going to have, so the first one is a set of outputs.
And here we're going to have a separate individual output.
So technically we're going to be using our neural network.
So once, the image and everything, like the values go
through the network from left to right over here,
they don't just spit out one set of values,
they spit up actually two sets.
And so the top set, we already know what it is,
it's, it's the possible actions
but here we're actually going to have another extra value.
So let's have a look at that.
How, what, what is that value?
So here we go.
That's the top.
So we just kind of like reduce the size
of this illustration.
The top output is are the Q values
as we discussed previously for the actions.
So they're, same thing, everything is same.
But then now this bottom part, oh
and the top part is actually called the actor.
We're going to give it a name.
That's the actor, because that's the part
where the agent chooses what it wants to do.
So that it's like it's acting, it's, it's as if it's
it's performing on stage.
And it'll make more sense
once we have the second name up on the screen as well.
And then the second output is just like one value
and that is V of S.
So that is the value of the state.
So if Q of S is the, Q of S and A,
is the Q value of a certain action.
And as you can see, that's why there's action one
action two, action three and up to action six,
so however many actions there possibly are in that state.
So in a given state S,
what is the Q value of taking action A?
A, action two, action one, action two, and so on.
Then here we're also predicting,
we're also using neural network to predict what is the value
of the state we're actually in.
And this part is called the critic.
And so that's the intuitive
or the kind of the not even full intuitive,
that's just like the start of the intuition
behind actor-critic that there's two outputs now
from the neural network, not just one.
Before, we just had that one output
which we now call the actor.
But now we have two outputs, actor and critic.
And there's gonna be a dynamic between them
which we'll explore further.
But for now, it's important to understand
that we are predicting not just the Q values
of the actions that you, that the agent can take
from this certain state.
But we're also predicting the value of being
in this current state using that same neural network.
So that's a core of the first step into the actor-critic.
And now we're going to need to talk about asynchronous
which we'll do in next tutorial
in order to understand exactly
what's going on between the actor-critic.
And the final thing for today is that all
of these Q values, as we know, that's also called policy.
So in some literature, in some blogs and some discussions
you might find in the actor-critic, you might find the
author talking about Q values on this side of the actor.
In some, in other literature and blog posts and discussions
you will find the, the author talking about the policy.
So, and usually it's used, they use a, like a Greek letter P
for representing the policy,
or it might just say policy of state.
So altogether, these are the policy
of database because as we remember, the policy is the
if you put all the actions together, the possible actions
and then it, it's deciding which action to take.
So it's, these are gonna be
like the probabilities of taking each action.
So that's the policy.
So don't, don't be thrown off,
if you see one or the other.
They basically mean the same thing.
So on one hand here, you've got the policy or the Q values,
on the other hand, you've got the actual value of the state
and they're being predicted from the neural network.
So that's the start of the actor-critic.
We'll continue with this in the next tutorial
when we're talking about asynchronous,
and I look forward to seeing you there.
Until then, enjoy AI.