In this lecture, we are going to start discussing modern art and units.
You may have heard of these already, so I'll just tell you right now what they are.
The Alice team and the guru Alice TM stands for long short term memory and group stands for gated recurrent
unit.
They are both essentially the same thing, with the argument to be a more simplified and efficient version
of the last year.
This lecture will discuss why the LSD Yemen's you are needed in the first place when the simple answer
seems pretty good already.
We'll also look at the underlying equations for the lithium and giu and importantly, a convenient perspective
for looking at them.
One mistake a lot of beginners make is that they see a bunch of equations and they get very intimidated.
They think, What the heck is this just a bunch of equations?
What do I do with this?
So hopefully this lecture will take you from that lazy beginner's mindset to how a proper machine learning
engineer would think about Elysium's, and she argues.
First, we need to discuss why we have these fancy new recurrent units in the first place.
Why is this simple Arnon unit not good enough to understand this?
We have to go back to the vanishing gradient problem.
As with our CNN, we can represent the output prediction y hat of Big T as a function of the inputs
x one x two all the way up to x t.
Once we do this, we can see that it's just a really big composite function.
Now suppose we want to take the derivative with respect to W X H, we can see that w x h appears several
times in this equation.
Now, the actual derivatives themselves and how to find them are not what we have to consider here.
But what is important is which derivatives are going to show up.
Well, if w x h appears multiplied by X of T, then we're going to have to find the derivative of W
x h times x of T at some point in our expression for the gradient of W x h.
It should be clear that W x h appears in front of all the XS for every time step.
So ultimately, all of these single derivatives are going to appear somewhere in our expression for
the gradient of our costs with respect to w x h.
But another thing that's important to consider is how deeply nested each of these terms is with respect
to why have of Big T.
What is clear is that the term involving X1 is the most deeply nested.
The term involving X2 is the second most deeply nested and so on.
Now what do we remember about the derivative of composite functions?
For my theory of aliens?
Well, we know that composite functions turn into multiplication in the gradient.
This is due to the chain rule of calculus.
Therefore, the more deeply nested you are, the more terms you have to multiply by when you're finding
the gradient.
In other words, aren't ends are particularly vulnerable to the vanishing gradient problem.
The farther back in the sequence the input is, the more vulnerable it is to the effects of the vanishing
gradient.
This is why it's often said that awnings have problems learning long term dependencies.
The answer simply can't learn from inputs that are too far back.
So how does the vanishing gradient problem manifest in actual machine learning applications?
Let's suppose we're doing natural language processing, and we would like to extract some information
about a document.
So to give you a concrete example, suppose our A.I. is reading the Wikipedia page about Albert Einstein.
It's read the entire page.
Now you ask your A.I. on what day was Albert Einstein born?
Unfortunately, a simple answer would not be able to answer this, even though this information appears
in the article.
The problem is it appeared at the beginning of the article.
So by the end of the article, the simple answer has simply forgotten what it had read earlier.
Now you might think, AHA, have a solution to this vanishing gradient problem.
It is to use real use.
Unfortunately, with Aunt Ends, things aren't so simple.
Of course, you can always try to use while using your aunt ends and verify experimentally what the
results are.
But in deep learning, we found that the most effective way of dealing with this problem is to use entirely
different units, namely the LTCM and the glue.
One interesting historical fact is that the system was created a very long time ago, 1997.
This is more than a decade before deep learning was even called deep learning.
So this is why I always say to people research interesting ideas.
Don't just go through your career chasing what is popular.
By doing that, you were like a hamster running on a spinning wheel, going nowhere.
If you're only chasing popular mainstream ideas, you would have never come across this idea of LACMA.
It's more than a decade old part of a feel that nobody even cares about.
Do what is interesting, not what is popular.
Now, the LSHTM is an extremely complex unit.
The guru was invented more recently in 2014, but it uses a lot of the same ideas.
So I always like to start with the GIU so that you can learn the principles in a simpler setting and
then go back to the LSHTM.
So what is the gated recurrent unit?
First, let's start with the basic principle from a simple aren't.
We still want to have some hidden state at time t h of T.
This hidden state will still depend on x of T the current input and h of T minus one the previous head
in state.
The only thing that will change is how each of T is calculated.
Now, there's a little bit of a debate on how to present the group analyst Yeah.
Is it more useful to look at the equations?
Or is it more useful to look at diagrams?
Now, if you are a beginner, you might automatically say pictures pictures are the best for learning.
Unfortunately for you, that's not what we are going to do a lot of poorly written blogs that all copy
from each other.
Use these same diagrams as if they were self-explanatory.
You look at this picture and all of a sudden, bam, you understand LATimes and gurus.
Personally, I find this doesn't work for me.
I know that other prominent lecturers feel the same way.
Of course, if you find these pictures useful, you're welcome to look at them and try to make sense
of them.
But for me and many others, I know it's actually the equations which are most useful.
That might seem backwards to a beginner who is very terrified of math, but hopefully you are past that
point right now.
I promise you that there's nothing here that we haven't seen before.
OK, so enough chitchat.
Where's the glue?
Well, here are the equations for the glue to sort of give you some perspective on this.
I've also included the equation for a simple, recurring unit so that you can compare the complexity
of the two.
The simple, recurring unit is essentially just one line.
Each of T depends on acts of T and H of T minus one.
The Giu is three lines because we have to calculate three different things.
First, we calculate Z of T, which is called the update gate vector.
Second, we calculate R of T, which is called the reset gate vector.
And finally, we calculate H of T, which is the head and state vector, which is as with the simple
recurrent unit, what gets passed on to the next layer in the neural network.
So let's analyze these equations so that we can make sense of the giu.
First, it's important to discuss shapes you want some visualization of the objects we're dealing with.
Well, it's helpful to know that all of these new vectors are just as vectors of size m the same size
as age of T.
As always, this is a hyper parameters, so you can choose how many features the hidden layers should
have.
This also implies the shape of all these weights.
If a weight is going from X of T to one of the gate vectors, then it must be of size D by M.
If a weight is going from H of T minus one to one of the gate vectors, then it must be of size m by
m.
And of course, since all the vectors are of size M, then all of the bias terms must also be of size
m.
Next, let's consider what the GRU is trying to do.
I think the names are very helpful here.
ZTE is the update gate victor in R.T. is the reset gate victor.
By the way, if you didn't know the circle with a dot in, it means element wise multiplication.
So one minus z of T is an m size vector and H of T minus one is also an m size vector.
When you multiply these together, you are doing element wise multiplication.
So let's start by looking at zero.
How is it used?
We can see the z of T gets multiplied by one term involving X of T and each of T minus one.
Then one minus z of T gets multiplied by the previous hidden state h of T minus one.
This is very helpful.
You can think of Z of T as telling us What should we do?
Should we take this new value for the head and state?
Or should we just remember the previous hidden state age of T minus one?
Why is that helpful?
Well, remember what we learned about the vanishing gradient.
If we use a simple aren't in its printed forgetting things that it's seen in the past?
By using this gate, it explicitly allows us to remember the previous head and state so that the head
and state can be carried forward to the next hidden state.
By the way, since Z of T is the output of a sigmoid, its values are always between a zero and one.
Therefore, for each of T, we are always taking a weighted sum of the previous H of C minus one.
And this other function.
So if zero is close to zero, then we'll remember the old value h of T minus one.
But if Z of T is close to one, then we'll take the new value and forget the old H of T minus one.
By the way, you can think of this other function as analogous to what the simple orient and is doing.
This is very important, but there's one more piece of the puzzle before we can move on to a helpful
interpretation of what's going on.
I want you to look very carefully at this equation for Z of T, what does this look like to you?
Well, it's a sigmoid of some dot products, plus a bias term.
What does that look like?
Well, you might recognize this as exactly like a dense layer.
Now you might object saying that there are two weights, so it's not exactly like a dense layer.
But in fact, the way that these units are often implemented, what we do is we concatenate X of T and
H of T minus one into a single vector.
Let's just call it V for sure.
If we concatenate those inputs and we concatenate the weights, then we just get back to our regular
old dense layer where we have one input multiplied by one weight matrix plus a bias term.
In other words, what is this?
This is just a neuron.
Now, notice that it also ends with a sigmoid.
Importantly, this sigmoid is not a hyper parameter.
This is usually always a sigmoid because the output is always a number between a zero and one.
That's how we can take Z of T and a one minus z later on in the G, are you?
So what is this neuron predicted?
It's like a binary classifier telling us essentially the probability that we should take the new value
for h of T, or if we should keep the old value of T minus one.
So if we ignore the reset gate for now, here's how we can interpret the GIU.
First, remember that each of T is essentially the weighted sum of two things each of T minus one in
the output of a simple aan n z of T.
The update gate vector is like an output probability, telling us the probability that we should keep
the output of the simple answer.
In other words, keep each of T minus one with probability one minus C of T and keep the simple aren't
an output with probability C of T.
The final H of T is then just a mixture of these two, allowing us to keep the most important parts
of each of T minus one and discard the rest.
Now, I want to make one small clarification.
Please note that this is really for intuition only.
These are not really probabilities of keeping and discarding because we never actually keep her discard.
Instead, what we actually do is take a mixture of the two components, for instance, 25 percent of
component one and 75 percent of component two.
But we don't actually keep or discard.
In actuality, the word gate is very helpful.
Imagine two openings side by side.
Now imagine a gate, which is the same size as each opening, but can only slide between the two openings.
Thus, it can only fully cover one of the openings at once, or it can simply cover a part of each.
But notice that in total, it always covers an area equal in size to one full opening.
In other words, if one opening is 25 percent clear, the other opening will be 75 percent clear and
they always seem up to 100 percent.
So picture this like an ice cream maker that can give you some mixture of chocolate ice cream and vanilla
ice cream.
Maybe you like chocolate ice cream more so you choose 90 percent chocolate and 10 percent vanilla.
Sliding the gates at the appropriate point to achieve this.
Note that in practice, this is called a convex combination.
You can also think of it like a soft mixture or a soft selection.
Let's now turn our attention to the other gate, Victor, the reset gate, victor.
How does this come into play?
Well, first, let's consider how this is calculated.
As you can see, it's just another neuron, so that should give you some comfort.
There is really nothing new here.
As I promised, we're just making use of neurons, which we've already learned about just neurons connected
to motor neurons.
Now it's called the reset gate vector, which is a good hint about what it's for.
Remember that we called the second part of the age of T calculation a simple answer, but this wasn't
entirely true.
This is because this is where the reset gate appears.
As you can see, the reset gate is used by doing an element wise multiplication with each of T minus
one.
Importantly, remember that the values of of T are always between zero and one.
What happens if we multiply each of T minus one by a value close to zero?
Well, it just gets closer to zero.
What if we multiply by one?
Then it just stays the same as it was before.
In other words, before multiplying by the a hidden way w h, we decide which parts of each of T minus
one we want to remember and which parts we want to forget.
So we forget them simply by resetting them back to zero.
Hence, the reset gate.
Note that this performs a very similar function to the update gaisie of tea.
It's just in a different place.
So it's just reinforcing our ability to remember and forget different parts of each of T minus one.
OK, so to summarize what we've learned so far.
What is the glue and why is it useful?
Are you?
First and foremost, has the same API as the simple, recurring unit that put is still HFT, and it
still depends on age of T minus one and 50.
The difference between this and the simple recurring unit is it has functionality for remembering and
forgetting what was in h of T minus one.
This solves a problem we had with Simple Rain ends where due to the vanishing gradient problem, it
would forget things that it saw earlier in a sequence.
We accomplished this by adding little binary classifiers in the form of logistic regression neurons
that output a prediction on whether to remember or forget.