In this lecture, we are going to start discussing modern art and units, you may have heard of these
already, so I'll just tell you right now what they are.
The LSM and the GROU.
ASTM stands for long and short term memory and Giuse stands for gaited recurrent unit.
They are both essentially the same thing with the GIU meant to be a more simplified and efficient version
of the LSM.
This lecture will discuss why the l'Est Yemanja are you are needed in the first place when the simple
answer seems pretty good already.
We'll also look at the underlying equations for the LSM and who are you and importantly a convenient
perspective for looking at them.
One mistake a lot of beginners make is that they see a bunch of equations and they get very intimidated.
They think, what the heck is this?
Just a bunch of equations.
What do I do with this?
So hopefully this lecture will take you from that lazy beginner's mindset to how a proper machine learning
engineer would think about Elston's and Giuse.
First, we need to discuss why we have these fancy new recurring units in the first place.
Why is a simple Orendain unit not good enough to understand this?
We have to go back to the vanishing gradient problem.
As with our CNN, we can represent the output prediction, we had a big T as a function of the inputs
x1 x2 all the way up texte.
Once we do this, we can see that it's just a really big composite function.
Now, suppose we want to take the derivative with respect to what age we can see that age appears several
times in this equation.
Now, the actual derivatives themselves and how to find them are not what we have to consider here,
but what is important is which derivatives are going to show up?
Well, if X H appears multiplied by X of T, then we're going to have to find the derivative of W H
times, X of T at some point in our expression for the gradient of x h, it should be clear that H appears
in front of all the is for every timestep.
So ultimately all of these single derivatives are going to appear somewhere in our expression for the
gradient of our costs.
With respect to x h.
But another thing that's important to consider is how deeply nested each of these terms is with respect
to why hat of Big T, what is clear is that the term involving X1 is the most deeply nested.
The term involving X2 is the second most deeply nested and so on.
Now, what do we remember about the derivative of composite functions for my theory of Andsnes?
Well, we know that composite functions turn into multiplication in the gradient.
This is due to the chain rule of calculus.
Therefore, the more deeply nested you are, the more terms you have to multiply by when you're finding
the gradient.
In other words, Arnon's are particularly vulnerable to the vanishing gradient problem.
The farther back in the sequence the input is, the more vulnerable it is to the effects of the vanishing
gradient.
This is why it's often said that Arnon's have problems learning long term dependencies, the answer
simply can't learn from inputs that are too far back.
So how does the vanishing Iranian problem manifest in actual machine learning applications, let's suppose
we're doing natural language processing and we would like to extract some information about a document.
So to give you a concrete example, suppose I is reading the Wikipedia page about Albert Einstein.
It's read the entire page.
Now you ask your eye on what day was Albert Einstein born.
Unfortunately, a simple answer would not be able to answer this, even though this information appears
in the article.
The problem is it appeared at the beginning of the article.
So by the end of the article, the simple answer has simply forgotten what it had read earlier.
Now, you might think, aha, I have a solution to this vanishing ingredient problem, it is to use
rellenos.
Unfortunately, with Arnon's, things aren't so simple, of course, you can always try to use while
using your own ends and verify experimentally what the results are, but in deep learning, we found
that the most effective way of dealing with this problem is to use entirely different units, namely
the LSM and the glue.
One interesting historical fact is that the LSM was created a very long time ago, 1997.
This is more than a decade before deep learning was even called deep learning.
So this is why I always say to people research, interesting ideas.
Don't just go through your career chasing what is popular.
By doing that, you were like a hamster running on a spinning wheel, going nowhere.
If you're only chasing popular mainstream ideas, you would have never come across this idea of Yepes.
It's more than a decade old part of a field that nobody even cares about.
Do what is interesting, not what is popular.
Now, the LSM is an extremely complex unit, the GIU was invented more recently in 2014, but it uses
a lot of the same ideas.
So I always like to start with the Guiyu so that you can learn the principles in a simpler setting and
then go back to the LSM.
So what is the gaited recurrent unit?
First, let's start with the basic principle from a simple arnet.
We still want to have some head in state at time t h of t this head and state will still depend on of
t the current input and each of T minus one, the previous head and state.
The only thing that will change is how each of T is calculated.
Now, there's a little bit of a debate on how to present that UNRWA, is it more useful to look at the
equations or is it more useful to look at diagrams?
Now, if you are a beginner, you might automatically say pictures, pictures of the best for learning.
Unfortunately for you, that's not what we are going to do.
A lot of poorly written blogs that I'll copy from each other use these same diagrams as if they were
self-explanatory.
You look at this picture and all of a sudden, bam, you understand Elston's and Giuse.
Personally, I find this doesn't work for me.
I know that other prominent lecturers feel the same way.
Of course, if you find these pictures useful, you're welcome to look at them and try to make sense
of them.
But for me and many others I know, it's actually the equations which are most useful that might seem
backwards to a beginner who is very terrified of math, but hopefully you are past that point right
now.
I promise you that there's nothing here that we haven't seen before.
OK, so enough chitchat.
Where's the group?
Well, here are the equations for the GIU to sort of give you some perspective on this.
I've also included the equation for a simple recurrent unit so that you can compare the complexity of
the two.
The simple recurring unit is essentially just one line.
HFT depends on activity and each of T minus one, the Gosu is three lines because we have to calculate
three different things.
First, we calculate CFT, which is called the update gate vector.
Second, we calculate gravity, which is called the reset gate vector.
And finally we calculate HFT, which is the head and state vector, which is, as with the simple recurrent
unit, what gets passed on to the next layer in the neural network.
So let's analyze these equations so that we can make sense of the Guiyu.
First, it's important to discuss shapes, you want some visualization of the objects we're dealing
with?
Well, it's helpful to know that all of these new vectors are just vectors of size m the same size as
H of T, as always.
This is a hyper parameters, so you can choose how many features the hidden layer should have.
This also implies the shape of all these weights.
If a weight is going from X 50 to one of the gate vectors, then it must be of size D Biem.
If weight is going from each of T minus one to one of the gate vectors, then it must be of size MBM.
And of course, since all the vectors are of size M, then all of the bias terms must also be of size
M.
Next, let's consider what the group is trying to do.
I think the names are very helpful here.
CFT is the update gate vector and AFTE is the reset gate vector.
By the way, if you didn't know the circle with a dot in, it means Element Y's multiplication.
So one minus Z is an M size vector and each of T minus one is also an M size vector.
When you multiply these together, you are doing Element Y's multiplication.
So let's start by looking at t how is it used?
We can see that zaftig gets multiplied by one term involving activity and each of T minus one, then
one minus Z gets multiplied by the previous hidden state, each of T minus one.
This is very helpful.
You can think of ZT as telling us what should we do?
Should we take this new value for the head and stay, or should we just remember the previous hidden
state state of T minus one?
Why is that helpful?
Well, remember what we learned about the vanishing gradient.
If we use a simple answer, it's parents are forgetting things that it's seen in the past.
By using this gate, it explicitly allows us to remember the previous hidden state so that the hidden
state can be carried forward to the next hidden state.
By the way, since ZT is the output of a sigmoid, its values are always between a zero and one.
Therefore, for each of T, we are always taking a weighted some of the previous H of T minus one and
this other function.
So if the left is close to zero, then we'll remember the old value of T minus one.
But if it is close to one, then we'll take the new value and forget the old H of T minus one.
By the way, you can think of this other function as analogous to what the simple AURIN and is doing.
This is very important, but there's one more piece of the puzzle before we can move on to a helpful
interpretation of what's going on.
I want you to look very carefully at this equation for CFT, what does this look like to you?
Well, it's a sigmoid of some DOT products, plus a bias term.
What does that look like?
Well, you might recognize this as exactly like a dense layer.
Now, you might object saying that there are two weights.
So it's not exactly like a dense layer.
But in fact, the way that these units are often implemented, what we do is we concatenate X of T and
H of T minus one into a single vector.
Let's just call it V for sure.
If we can coordinate those inputs and we concatenate the weights, then we just get back to our regular
old dense layer where we have one input multiplied by one weight matrix plus a bias term.
In other words, what is this?
This is just a neuron.
Now notice that it also ends with a sigmoid.
Importantly, this sigmoid is not a hyper parameter.
This is usually always a sigmoid because the output is always a number between a zero and one.
That's how we can take Z of T and one minus CFT later on in the Q.
So what is this neuron predicting?
It's like a binary classifier telling us essentially the probability that we should take the new value
for each of T or if we should keep the old value of T minus one.
So if we ignore the reset gate for now, here's how we can interpret the GIU first, remember that each
is essentially the weighted sum of two things, each of T minus one and the output of a simple R zaev
t the update gate vector is like an output probability, telling us the probability that we should keep
the output of the simple line.
In other words, keep each of T minus one with probability one minus C of T and keep the simple on and
output with probability of T.
The final activity is then just a mixture of these two allowing us to keep the most important parts
of each of team minus one and discard the rest.
Now, I want to make one small clarification, please note that this is really for intuition only.
These are not really probabilities of keeping and discarding because we never actually keep or discard.
Instead, what do we actually do is take a mixture of the two components.
For instance, twenty five percent of component one and seventy five percent of component, too.
But we don't actually keep or discard.
In actuality, the word gate is very helpful.
Imagine two openings side by side.
Now, imagine a gate which is the same size as each opening, but can only slide between the two openings.
Thus it can only fully cover one of the openings at once, or it can simply cover a part of each.
But notice that in total, it always covers an area equal in size to one full opening.
In other words, if one opening is twenty five percent clear, the other opening will be seventy five
percent clear and they always sum up to one hundred percent.
So picture this like an ice cream maker that can give you some mixture of chocolate ice cream and vanilla
ice cream.
Maybe you like chocolate ice cream more so you choose 90 percent chocolate and 10 percent vanilla sliding
the gate to the appropriate point to achieve this.
Note that in practice, this is called a convex combination.
You can also think of it like a soft mixture, a soft selection.
Let's now turn our attention to the other great victory, the reset gate, Victor, how does this come
into play?
Well, first, let's consider how this is calculated.
As you can see, it's just another neuron, so that should give you some comfort, there is really nothing
new here.
As I promised, we're just making use of neurons, which we've already learned about, just neurons
connected to more neurons.
Now it's called the resegregate vector, which is a good hint about what it's for.
Remember that we called the second part of the calculation a simple answer, but this wasn't entirely
true.
This is because this is where the Redgate appears.
As you can see, the reset game is used by doing an element Y's multiplication with each of T minus
one.
Importantly, remember that the values of our of T are always between zero and one.
What happens if we multiply each of T minus one by a value close to zero?
Well, it just gets closer to zero.
What if we multiply by one?
Then it just stays the same as it was before.
In other words, before multiplying by the headwinds ahead and wave H, we decide which parts of each
of T minus one we want to remember and which parts we want to forget.
So we forget them simply by resetting them back to zero, hence the reset gate.
Note that this performs a very similar function to the update gazy of T.
It's just in a different place.
So it's just reinforcing our ability to remember and forget different parts of T minus one.
OK, so to summarize what we've learned so far, what is the goal and why is it useful?
Ajamu, you first and foremost has the same API as the simple recurring unit.
The output is still HFT and it still depends on each of T minus one index of T.
The difference between this and the simple recurring unit is it has functionality for remembering and
forgetting what was in each of T minus one.
This solves a problem we had with simple Arnon's where due to the vanishing gradient problem, it would
forget things that I saw earlier in a sequence.
We accomplish this by adding little binary classifiers in the form of logistic regression neurones that
I'll put a prediction on whether to remember or forget.