-: Hello and welcome back to the course
on artificial intelligence.
In today's tutorial,
we're going to talk about an add-on
that we're going to be implementing for our A3C algorithm.
It's called the long short term memory,
or the LSTM for short.
Let's have a look at what we have so far,
and then we'll discuss why we need the LSTM
and what an LSTM is.
So far, we've discussed the A3C algorithm,
we've talked about all three letter A's in the A3C,
and of course, we've seen that it's actually a bit more,
or much more complex than we have on this image,
we actually have three or more multiple agents
going through the environment and they're communicating
between each other and so on,
but for simplicity's sake, for today's tutorial,
we're going to just illustrate everything
with this one agent.
At the end, we have this extra part, the critic part.
Basically, once we have a state, that state, or this image,
goes through a convolutional layer,
then it goes through a pooling layer,
it goes through a flattening layer,
and at this point, we have values or numbers
which are then propagated through the network
and they go into the hidden layers,
and then as the output, we get the policy,
or the actor part, and then get the value of the state,
or we get the critic part.
And what we're going to do today
is we're going to talk about this hidden part.
In the hidden layers,
we can actually take it to the next level
and we can add a modification.
And we've already seen that multiple modifications exist
for the A3C algorithm.
We've seen one of them, we've seen that,
in some cases, you can have this main part of the network,
which is individual to every agent,
or you can have this main part of the network,
which is shared, and that's what we saw that
in our previous Intuition tutorials,
we had a shared part of the network,
this network was shared between the agents,
and as Atlan will tell you more in the practical tutorials,
that really helps with the challenge of Breakout.
And there are lots and lots of lots of other ways
you can modify the A3C algorithm,
lots of other additions that can be implemented.
And one of them we're going to discuss,
because we are actually going to have it
in the practical side of tutorials,
here, before your hidden layers,
what you can add is an LSTM layer,
a neural network layer, which allows your A3C algorithm,
which allows your algorithm to have memory,
which allows the algorithm to remember what happened before.
And we'll talk about LSTM just now in more detail.
But basically, you can add an extra layer here,
which is the LSTM layer, and enhance your algorithm
with some additional memory of another feature.
And what you'll actually see in the practical tutorials
is that we didn't even need any hidden layers
after the LSTM layer.
You'll see that in Atlan's implementation,
he's got the flattening layer right away,
after that, he's got the LSTM layer,
so basically, this box represents the LSTM layer,
and then after that, right away you've got the output.
You didn't even need any other hidden layers after that,
simply because that's how much power the LSTM layer
adds to the algorithm.
And again, the algorithm,
or the architecture of your neural network,
it's a very individual thing, it's a personal preference,
it's a very creative thing,
so you might want to have two LSTM layers,
you might have one LSTM layer
and then several, like five hidden layers after the LSTM
That's totally up to you
and for you to experiment and explore.
But this is what we came up with in the practical tutorials.
You'll see that we have a flattening layer,
and after that, we've got an LSTM layer,
and then the output.
Now that we've talked about the LSTM layer so much,
what is this LSTM layer?
Well, the LSTM layer adds memory,
gives a feature, like allows the neural network
to have memory about what happened
in the previous situations,
and it is often symbolized or shown with a symbol
which looks like this.
This is us just getting started into it,
and I'm just putting here, I know it looks very crooked,
but I'm putting it here so you can see
when we further discuss this image,
you can see what's going on.
The output of this layer goes into here,
and that is our, so this is a whole layer going in here,
so it's a vector of values, X is a vector of values,
goes into our LSTM, which we'll just discuss this now,
and then as an output, you get another vector,
which is the concatenation of this store,
or somehow it ties in with network, in our case,
as an output, you get this and you get this.
Let's have a look at this in more detail.
We're just gonna focus on this part.
In fact, we're going to,
as you probably noticed by the letters being on this side,
we're gonna turn it to the side, so like that,
and this whole like jumble around
was just to reiterate the fact
that even though it looks like this,
what is actually happening is a layer of values,
a whole vector of values, is going in here,
something is happening, which we'll discuss just now,
and then a whole vector of values is going on here.
This is the layer, it's not just one element of it,
this is the layer itself.
Let's go back, again, just to reiterate.
Layer goes into this layer, something happens,
layer comes out.
That's the LSTM, it's just on its side.
It's just easier to discuss this way,
and that's the common representation.
All right, now that we agree why this image was on its side
and how we're going to proceed with this,
let's start digging into this LSTM situation a bit more.
What happens inside the LSTM layer?
This is what it looks like,
and of course, this looks very complex,
and we're definitely not gonna go
through all of this right now
simply because there's quite a lot to discuss.
These are point-wise operations,
these are layer-wise operations,
and there's just a lot going on,
or a lot of intricate details,
which we're not going to go into
because otherwise, it would blow out this course.
And the purpose is not to talk about LSTMs here,
we're just gonna utilize these LSTMs.
And if you'd like to learn more about LSTMs,
you can either go to over here, Christopher Olah's blog,
he's got a good description of LSTMs,
or we also talk about LSTMs in our deep learning HSN course.
You can check it out there.
We've got a whole section on recurrent neural networks
and LSTMs, as well.
Basically, this is the internal part of the LSTM.
And what happens is like the layer goes in,
so we're gonna talk about this on an intuitive level,
on a very basic intuitive level,
just what is gonna be enough for us to understand
what happens, why there's memory,
and so that you can also better understand
what Atlan is talking about when he's implementing this.
A layer goes in, all of this,
something basically goes on here, a layer goes out.
What we need to actually see is that these parts,
there's actually additional inputs into this layer.
Remember this, usually you have like an input
from a previous layer, then this layer,
and then you have an output,
if you think of that image we had previously,
the normal network, which is not on its side,
which is like from left to right, not from bottom to top.
But with an LSTM, you actually have more inputs.
I know it's getting even more complex.
But these things, at least we can understand them.
This is your memory cell, this is the key,
and this is what you'll hear Atlan talk about.
The memory cell is something that is saved in the LSTM.
These inputs and these outputs, they're actually,
here what you're looking at is the time access.
This is unraveled in time.
In one specific iteration, this happens,
but then this value is taken from the past
and this value is passed into these values,
these values are taken from the past,
and these values are passed onto the future.
And how they pass, well it's through the way the LSTM works.
Without worrying about too much what's going on here,
all we need to understand is that the layer goes in,
and here we already have a value which came from the past,
which is stored inside the LSTM,
inside the long short term memory,
we've got this memory cell,
and whatever value was here before, it just stays here.
As you can see, it just goes through,
it flows through freely,
except for these point-wise operations
where it can be either closed off
or something can be added to it.
But regardless of that,
it's just some value that flows through freely.
It's basically, it's passed on to the next point in time,
the next point in time,
so you can just think of it as like some memory
that's like a flash drive or something like that
that this cell has.
And so, it just remembers the previous value
of this was in here,
and then it can use that to either add to it
or read from that value and so on.
And this value is the hidden state.
It's the, so hence the h,
and the hidden state is basically another value
that comes from the past and then is used inside the LSTM.
And as you can see, at the end, after all of this happens,
what you get is you get a layer that comes out
and it is, so you get this value that comes out,
and it is the same value that's passed forward.
Basically, the LSTM remembers two things.
There's a constant value that is just like stays in the LSTM
that can be changed, like there's a flash drive
for like a constant value, so the memory cell.
And so, you have the luxury of storing something
in that space, in that memory,
and it'll be passed onto the future.
Whenever, in the next iteration,
so like the algorithm was in an environment,
it saw something, it did something, and so on,
and then in the LSTM, you can store a certain value,
then it will remember this value
even when it's in the next state.
And also, the other value that it'll remember,
it'll remember its previous output,
it automatically will remember its previous output,
so the output goes here and goes here.
That's basically the very, very very high level
of what happens in an LSTM.
Once again, if you'd like more details,
there's lots of resources where you can find,
and at this stage,
we just don't need to go into that much detail
on all of these things.
We just need to understand what a memory cell is,
what a hidden state is,
and how they facilitate memory for the LSTM.
And the question is,
so now that we kind of have a general overview
of all of this, to reinforce or to solidify this knowledge,
or kind of like give a reason for this knowledge,
let's ask the question, why do we need memory?
Why do we need memory in our A3C or other algorithms?
Well, let's look at our example,
the challenge that we're taking on in the section.
The challenge is Breakout, and what happens in Breakout?
Well, in Breakout, you've got this environment,
these little blocks that you need to destroy
with this little ball, and you need to make sure
that this is your kind of like racket or platform
that's moving around, and it must,
wherever the ball is flying, it must catch the ball
and it'll bounce off the platform
and it'll go back and hit,
it'll bounce off the walls as well,
and go back, hit a block, and come back.
And so, that's the essence of what you need to accomplish.
But now, let's look at this ball.
Imagine you are an A3C algorithm,
or an agent inside, one of those agents inside A3C.
You see this picture.
What do you extract from here?
What would your actions be here from here?
You can see the ball is flying, right?
Well, it's flying, right, so it's going somewhere.
And maybe it's flying towards you, right?
Could you make this conclusion?
Could you like anticipate that it's coming towards you?
You probably could and maybe you're in the right spot
to catch the ball, but what if the ball
is actually not flying that way,
what if it's flying that way, what if it's flying that way?
The thing is, you cannot tell from this one image
which way it's flying because you don't know
where it was in the previous moment of time.
If it was here, then it's flying this way.
If you knew the previous moment in time,
if you knew that it was here,
then now you know here is, as a human,
you just draw a line from these two and you'd be like,
oh cool, so it's going this way.
But if you knew it was here,
you draw a line for these and know it's going this way.
Moreover, look at this,
it could have actually been somewhere like over here.
Maybe it's going up, maybe it's actually going that way.
Maybe it was here and now it's going upward.
Just from the one image, it's very hard,
it's actually impossible,
it's just like geometrically impossible
to tell which way the ball is flying.
And so, that's why the LSTM, the memory,
actually really, really helps the algorithm.
Without the memory, you can still do a good job.
You could probably like guess
or find other ways of understanding where to go.
But with the LSTM, with just even that one memory,
so if we go back, even with that one value,
that was kind of like the output of the previous value,
or maybe you can store it here, or based on this value,
like based on the information that it gets
from the previous point in time,
so let's say from what happened here,
so that's where your ball was before,
so you can parse out information about the environment
from the previous point in time through here,
then now you have it.
Now, not only have your information from the image,
unless if we go back even further,
you'll remember that information from the image,
well this is Doom,
but we're actually working with Breakout,
information from the image came here, here, here,
turned into these flattened values.
And so, that's information from the image
coming into the LSTM.
And now, all of a sudden,
as you remember, coming not from somewhere,
but from the previous point in time,
so that's why you kind of actually demonstrate
it's coming from the top or from the bottom, left, right,
it's actually, it just stays in the LSTM layer,
you have that information,
just through the architecture of the LSTM,
you have the information about what happened previously.
And so, we go back,
that information here helps you now
make a decision on what to do,
helps the algorithm make a decision.
And now all of a sudden, it knows that,
oh okay, so the ball is actually flying in either area,
let's say it's flying in this direction
or in this direction, so I'm in the right place,
so I should stick around here,
the ball is coming in my direction.
Or if it realizes that the ball is flying there,
it should start moving to the left,
because if it waits a bit longer,
it'll be too late and it'll miss the ball.
Basically, that's how the LSTM layer really helps
in this algorithm, and that's exactly what we will see
when you're doing the practical tutorials with Atlan.
So there we go, that's how LSTMs work.
And just one additional note, as we mentioned at the start,
LSTMs are not a hundred percent necessary.
They're not a complete, they're not completely attached
to the A3C algorithm.
You might want to have them in the A3C algorithm,
you might not want to have them, depending on the situation,
depending on the architecture you choose.
There are lots of additions,
and we've already discussed the addition or the modification
where the neural network is shared between actors,
or is not shared between agents or not now the less LSTM.
There's another one that you'll see
in the practical tutorials
where we add entropy,
which is calculated through our policy loss,
and Atlan will walk you through that.
Basically, there's lots of different modifications
that can happen in an A3C algorithm.
Just remember that it depends on what you want to achieve
and it's also something that we'd encourage you to explore
if you're going to be implementing lots of these
and trying out different algorithms.
We've already discussed a couple,
and maybe you can find some additional modifications
that might be of interest to you.
Or maybe when you're watching these tutorials,
maybe by then more modifications have come out
which are very interesting.
Definitely that's something that you could look into
and that could further enhance your knowledge
of artificial intelligence
and how to create these algorithms.
And on that note, I hope you enjoyed today's tutorial
and I'll look forward to seeing you next time.
Until then, enjoy AI.