In this lecture, we are going to continue our discussion of the LSM and GIU, previously we discussed
the GIU and now we are going to move on to the Alstrom to recap, what did we learn about the GIU?
First we learn that the simple Arnon unit by itself is problematic because it can't remember long term
dependencies.
The way we solve this is we make the head and state the way that some of the previous head and state
and the new value for the state, which is essentially the same calculation as the symbol or end unit
in this way that you always has the opportunity to remember the previous and instead state allowing
it to carry that state forward in time, reducing the chance of forgetting it later.
Importantly, we recognize that these gates, which decide whether we should remember or forget, are
just binary classifiers.
There are little logistic regression neurons, something we are very comfortable with.
Now we are going to move on to the LSM, which is a little more complicated, but follows the same idea.
Interestingly, when Giuse first came out, the initial research suggested that there was no clear winner
between the two.
It was reported that the group performed comparably with the LDM.
This made the GIU a great choice because it has less parameters and therefore is more performant than
the allaster.
These days, the current belief is that this isn't true and that the LSM actually ended up being better.
In fact, at the time I am making this course, this latest research only came out about a year ago,
which is actually two years after I made my first nine Encore's.
So that just goes to show how fast things move in this field.
Experimental results are always changing our thinking about best practices and which techniques to choose
over others.
This also is a great example of Mirrool machine learning, experimentation and not philosophy.
I know for a lot of beginners, the approach they take is a philosophical approach.
They would try to use logical reasoning to deduce which is better.
The LACMA, the G are you the SDM has these Maritz, the YOU has these merits and so on.
Such things are useless.
Please don't waste your time on philosophy instead follow the example that these researchers have set
and use experiments to decide which is best.
There are two papers I've attached an extra reading that text that you'll want to check out.
The first is called on the Practical Computational Power of Finite Precision Arnon's for language recognition.
And the second is called massive exploration of neural machine translation architectures.
They both conclude that the LSM outperforms Giuse.
And so given the choice, I would probably choose the LSM by default.
So how does the system work?
Basically, you can think of it as like the Gyoo but with more state vectors and more gates, again,
it's my preference to understand that Elysium through the equations.
But if you find these visualizations helpful, then you can feel free to look at that as well.
The first difference with the GIU and the storm is that the storm has two states in addition to HFT,
which we call the head and state.
We also have CFT, the cells that sometimes the cell state is considered an additional head and state.
So you actually might pass both of these on to the next layer.
For us, we will usually ignore the cell state so you can think of it more like an intermediate value
that you calculate, just like the gate vectors.
So you calculate it, but you don't actually use it to pass on to the next layer.
So even though the LSM has an extra step, you can still think of it as having almost the same API as
a simple AURIN in energy.
Are you?
It still outputs the final hit and state of Big T.
It's just that for the inputs, in addition to acts of one all the way up to acts of Big T and the initial
head and state H of zero.
We also have the initial cells they see of zero for the outputs.
It's a little strange due to how high torch works, basically will get H of big T and C of Big T, but
the details might be surprising, but we'll discuss that more later in this lecture.
OK, so let's get to the LSM calculations.
I want to remind you that while this might seem complicated, it's not don't be scared by the sight
of equations, but instead try to see the forest for the trees.
Each of these is just a neuron.
Once you start to realize that each thing here is a binary classifier or logistic regression, it becomes
a lot simpler to interpret.
So what do we see?
First, we see that there are more of these gates.
Specifically, we have F.T. the forget gate vector.
Next we have IMT, the input gate vector, also known as the update gate vector, analogously to the
Jiahu.
And next we have all of T the output gate vector.
So hopefully this is the easy to remember because the word forget starts with f the word input starts
with I and the word output starts with, oh, next we have the cell state CFT.
You might notice that this actually takes on the role of HFT from the chair.
You specifically, we have the weighted sum of two terms.
The first term is just the previous cells, the C of T minus one.
How much of this we end up keeping is controlled by the forget game F of T?
So control is how much of C of T minus one we want to forget.
The second term is the simple answer.
How much of this we end up keeping is controlled by the input gate.
I have t and notice that this has exactly the same equation as the simple answer.
Unlike the G are you.
Now note that for the simple answer.
We have the activation function F of C this can be anything but usually by default is the Tanach.
Finally we have the hidden state each of T, which is just a simple transformation on the South State
CFT in particular, we apply another activation function F of H, which also can be anything but by
default, it's also usually a teenage.
We then multiply this by the output of T.
So the output gap controls which values of the cell state we actually pass through to the head and state
each of T.
Note that in PI, it's not possible to choose FFC enough of each individually and in fact, you can't
even change them from the default, as mentioned before, because Arnon's are so complicated, they're
usually programmed all in one module for the sake of efficiency, and they're not really composable
like the other modules we've learned about.
Of course, if you were writing your own a.T.M, you could customize this any way you wanted, but its
performance would most likely be suboptimal.
So you can see that by building up our knowledge first through simple Arnon's, then Giuse, then storms,
we can make something complex like the LSM seem very simple.
It's nothing but three logistic regressions or neurons to give us the forget, get the input gate and
the output gate.
Then the cell state is the weighted some of the previous cells day and the simple an weighted by the
forget gate and the input gate.
And remember, we want to offer the cell state the possibility to remember its old state so that we
can learn long term dependencies.
Finally, the hidden state HFT is just to squash the version of the cells there with the output gate
controlling which values are allowed to pass through.
By the way, I say squashed because we use the Tanach activation by default and it's not possible to
change.
So this is really not a hyper parameter you have to play around with.
In terms of code, how do the G are you in Elysium work as promised?
These have the same or similar API as the simple answer.
So if all you wanted to do is plug in a G or you are LSM in place of a simple or an end, it can be
almost as simple as changing the name.
Let's start with how to instantiate a group and list here.
As you can see, it's almost exactly the same as a simple answer.
We pass on a number of input features d the number of hidden features M and the number of Oranje and
layers the stack l.
I also pass in batch first equal to true so that all my data is always end by something instead of having
the end dimension somewhere else.
The crucial difference is that in a group analysts, there is no non-linearity argument because it's
not possible to change.
Let's not talk about how the GIU and LSM will be used in the forward function.
First, let's discuss the initial hit and state and what to pass into the Arnet.
Conveniently, the Jiahu has the exact same interface as the simple answer as the input we pass in the
data into the first argument and the initial stage, not into the second argument.
As before, the initial stage not is of shape L by N by M number of layers, by number of samples,
by a number of features for the SDM, we also require an initial state, luckily because the sun has
the same dimensionality as the hidden state.
We will also initialize this to L by N by M..
Now, let's talk about how to interpret the outputs from the Jiahu and LASU, as you recall, Arnon's
and PAE to return to things, not just one with the Elmina and we get one each, which is of shape and
by T by M, which tells us the hidden state at the final orendain and layer for each sample, each timestep,
any feature.
The second H is of shape l by and by m which tells us the hidden state at the final time.
Step for each sample, each RNA layer, any feature.
In the Gyoo It works exactly the same.
In the storm, we have something similar, but not the same.
The first thing that has returned is the same as what we get with the Alman unit, the hidden value
at each sample, each time step any feature.
The second thing is different.
We get the hidden state H and the cells they see, but only at the final timestep.
So this second age will be of shape by and by M, which tells us the hidden state at each sample, each
layer, any feature, we also get the cells of the same shape, which means it's the cells stay at each
sample, each layer and each feature.
Unfortunately, I'm unaware of how you would get the cells states at different times steps.