In this lecture, we are going to continue our discussion of the LSM and GIU, previously we discussed
the GIU and now we are going to move on to the Alstrom to recap, what did we learn about the GIU?
First we learn that the simple Arnon unit by itself is problematic because it can't remember long term
dependencies.
The way we solve this is we make the head and state the way that some of the previous head and state
and the new value for the state, which is essentially the same calculation as the symbol or end unit
in this way that you always has the opportunity to remember the previous and instead state allowing
it to carry that state forward in time, reducing the chance of forgetting it later.
Importantly, we recognize that these gates, which decide whether we should remember or forget, are
just binary classifiers.
There are little logistic regression neurons, something we are very comfortable with.
Now we are going to move on to the LSM, which is a little more complicated, but follows the same idea.
Interestingly, when Giuse first came out, the initial research suggested that there was no clear winner
between the two.
It was reported that the group performed comparably with the LDM.
This made the GIU a great choice because it has less parameters and therefore is more performant than
the allaster.
These days, the current belief is that this isn't true and that the LSM actually ended up being better.
In fact, at the time am making this course.
This latest research only came out about a year ago, which is actually two years after I made my first
nine Encore's.
So that just goes to show how fast things move in this field.
Experimental results are always changing our thinking about best practices and which techniques to choose
over others.
This also is a great example of Mirrool machine learning, experimentation and not philosophy.
I know for a lot of beginners, the approach they take is a philosophical approach.
They would try to use logical reasoning to deduce which is better.
The LACMA, the G are you the SDM has these Maritz, the YOU has these merits and so on.
Such things are useless.
Please don't waste your time on philosophy instead follow the example that these researchers have set
and use experiments to decide which is best.
There are two papers I've attached an extra reading that text that you'll want to check out.
The first is called on the Practical Computational Power of Finite Precision Arnon's for language recognition.
And the second is called massive exploration of neural machine translation architectures.
They both conclude that the LSM outperforms Giuse.
And so given the choice, I would probably choose the LSM by default.
So how does the system work?
Basically, you can think of it as like the Gyoo but with more state vectors and more gates, again,
it's my preference to understand that Elysium through the equations.
But if you find these visualizations helpful, then you can feel free to look at that as well.
The first difference with the GIU and the storm is that the storm has two states in addition to HFT,
which we call the head and state.
We also have CFT, the cells that sometimes the cell state is considered an additional hidden state.
So you actually might pass both of these on to the next layer.
For us, we will usually ignore the cell state so you can think of it more like an intermediate value
that you calculate, just like the gate vectors.
So you calculate it, but you don't actually use it to pass on to the next layer.
So even though the LSM has an extra step, you can still think of it as having almost the same API as
a simple AURIN in energy.
Are you?
It still outputs the final head and state each of Big T.
It's just that for the inputs, in addition to X of one all the way up to X of Big T and the initial
head and state H of zero.
We also have the initial cells they see of zero optionally the LSM unit can output the cells states,
but we'll usually ignore this.
OK, so let's get to the LSM calculations, I want to remind you that while this might seem complicated,
it's not don't be scared by the sight of equations, but instead try to see the forest for the trees.
Each of these is just a neuron.
Once you start to realize that each thing here is a binary classifier or logistic regression, it becomes
a lot simpler to interpret.
So what do we see?
First, we see that there are more of these gates.
Specifically, we have F.T. the forget gate vector.
Next we have iev t the input gate vector, also known as the update gate vector, analogously to the
Jiahu.
And next we have all of T the output gate vector.
So hopefully this is the easy to remember because the word forget starts with f the word input starts
with I and the word output starts with, oh, next we have the cell state CFT.
You might notice that this actually takes on the role of each of T from the you specifically.
We have the weighted sum of two terms.
The first term is just the previous cells, the C of T minus one.
How much of this we end up keeping is controlled by the forget gate F of T.
So control is how much of C of T minus one we want to forget.
The second term is the simple answer.
How much of this we end up keeping is controlled by the input gate.
I have t and notice that this has exactly the same equation as the simple answer, unlike the GIU.
Now, note that for the simple answer, we have the activation function f of see, this can be anything,
but usually by default, it's the Tanach.
Finally, we have the hidden state, each of T, which is just a simple transformation on the substate
CFT in particular.
We apply another activation function F of H, which also can be anything but by default, it's also
usually a teenage.
We then multiply this by the output of T.
So the output gap controls which values of the cell state we actually pass through to the head and state
H of t.
Not that intensive care carries, it's actually not possible to choose FCF of each individually, of
course, if you writing your own ASTM, you could customize this any way you wanted, but for tends
to flow and carries.
This is controlled by the activation argument, as usual.
So if you set activation to value, then both of these would be changed to value.
So you can see that by building up our knowledge first through simple Arnon's, then Giuse, then storms,
we can make something complex like the LSM seem very simple.
It's nothing but three logistic regressions or neurons to give us the forget, get the input gate and
the output gate.
Then the cell state is the weighted some of the previous cells there and the simple aren't weighted
by the forget gate and the input gate.
And remember, we want to offer the cell state the possibility to remember its old state so that we
can learn long term dependencies.
Finally, the hidden state of tea is just to squash the version of the there with the output gate controlling
which values are allowed to pass through.
By the way, I say squashed because most of the time we will use the default tannish activation.
So this is really not a hyper parameter you have to play around with.
Just choose the tannish by default.
As a side note, if you change the activation, you're LSM unit won't use the GPU, at least in this
version of Tenzer flow.
There are some other requirements too for the LSM to be GPU compatible.
So you want to check out that sense flow documentation for the most up to date list.
In terms of code, how did that go, you and ASTM work as promised?
These have the same API as the simple answer.
So if all you want to do is plug in Ajamu or LSM in place of the simple or an end, it's as simple as
changing the name.
In other words, just type in a different name and all of a sudden you have a more powerful model.
Easy.
Later on, there will be some useful options for these Arnon units that we can make use of.
I want to briefly mention these to you now so that, you know, they exist, although we won't be making
use of them right away.
First, recall that for each input, X of one of two all the way up to X, a big T. We will calculate
ahead in state H of one age of two, all the way up to H of Big T..
Currently we've seen that all the RNA units, the simple print in the GOP analysts only return at this
final hit and stay value H of Big T.
But there are scenarios where we would want the other head and states to so that we can calculate the
corresponding output predictions, why have one?
Why out of two and all the way up to Y had a big T.
In order to do that, it's very simple, all we need to do is pass in the argument return sequences
equals true.
When you do this, the output of the Arnett's unit will be of shape and by T, by M, because remember,
the hidden feature vector is of size M, and then when you apply a dense layer on top of this, your
final output is of shape and by t by K where K is the number of output nodes.
So you have an output for every timestep, for every sample.
Another interesting argument is the return state argument for the simple answer, and Ajamu, you this
is a bit superfluous.
We already know that the output is the hidden state.
So when we use the functional API and we grab the return value, this is already the hidden state.
So if you set returns, that equals true.
You'll just get back the same thing.
Return twice for the LSM.
The cell state is also considered a state.
So if you set returns, that equals true.
For the LSM.
It'll return three things the output, the hidden state and the cell state, the output and the head
and state are still the same thing, but the cell state is different.
Again, this is kind of random information at this point since we won't be using it immediately.
But it's good to have in the back of your mind just in case.