In this lecture, we are going to continue our discussion of the lithium and giu previously we discussed
the GIU, and now we are going to move on to the lithium.
To recap, what did we learn about the GIU?
First, we learn that the simple ANA unit by itself is problematic because it can't remember long term
dependencies.
The way we solve this is we make the head and state the weighted sum of the previous head and state
and the new value for the state, which is essentially the same calculation as the symbol or an end
unit.
In this way, the GIU always has the opportunity to remember the previous hidden state, allowing it
to carry that state forward in time, reducing the chance of forgetting it later.
Importantly, we recognize that these gates, which decide whether we should remember or forget, are
just binary classifiers.
There are little logistic regression neurons, something we are very comfortable with.
Now we are going to move on to the lithium, which is a little more complicated, but follows the same
idea.
Interestingly, when Jir use first came out, the initial research suggested that there was no clear
winner between the two.
It was reported that the GIU performed comparably with the lithium.
This made the GIU a great choice because it has less parameters and therefore is more performant than
the last year.
These days, the current belief is that this isn't true and that the lithium actually ended up being
better.
In fact, at the time, I'm making this course.
This latest research only came out about a year ago, which is actually two years after I made my first
nine encores.
So that just goes to show how fast things move in this field.
Experimental results are always changing our thinking about best practices and which techniques to choose
over others.
This also is a great example of my rule.
Machine learning, experimentation and not philosophy.
I know for a lot of beginners, the approach they take is a philosophical approach.
They would try to use logical reasoning to deduce which is better.
The LACMA, the gee are you.
The less GM has these merits, the GIU has these merits and so on.
Such things are useless, please don't waste your time on philosophy.
Instead, follow the example that these researchers have set and use experiments to decide which is
best.
There are two papers I've attached an extra reading text that you'll want to check out.
The first is called on the practical computational power of finite precision and ends for language recognition.
And the second is called massive exploration of neural machine translation architectures.
They both conclude that the STM outperforms joyeux use, and so given the choice, I would probably
choose the lithium by default.
So how does the election work?
Basically, you can think of it as like the GIU, but with more state vectors and more gates, again,
it's my preference to understand the LSM through the equations.
But if you find these visualizations helpful, then you can feel free to look at that as well.
The first difference with the GIU and the LSE team is that the LSHTM has two states in addition to HFT,
which we call the hidden state.
We also have CFT, the SLC.
Sometimes the sell state is considered an additional hit in state.
So you actually might pass both of these on to the next dense layer.
For us, we will usually ignore the cell state, so you can think of it more like an intermediate value
that you calculate, just like the gate vectors, so you calculate it, but you don't actually use it
to pass on to the next layer.
So even though the STM has an extra step, you can still think of it as having almost the same API as
a simple origin in energy, are you?
It still outputs the final hidden state of Big T.
It's just that for the inputs, in addition to X of one all the way up to X of Big T and the initial
hit and state each of zero.
We also have the initial cell state C of zero.
Optionally, the lithium unit can output the cell states, but we'll usually ignore this.
OK, so let's get to the lithium calculations.
I want to remind you that while this might seem complicated, it's not.
Don't be scared by the sight of equations, but instead try to see the forest for the trees.
Each of these is just a neuron.
Once you start to realize that each thing here is a binary classifier or logistic regression, it becomes
a lot simpler to interpret.
So what do we see?
First, we see that there are more of these gates.
Specifically, we have F50 the Forget gate vector.
Next, we have I have T the input gate vector, also known as the update gate vector.
Analogously to the grou.
And next, we have O of T the output gate vector.
So hopefully this is easy to remember because the word forget starts with f, the word input starts
with I and the word output starts with O.
Next, we have a cell state C of T.
You might notice that this actually takes on the role of H of T from the G you specifically.
We have the weighted sum of two terms.
The first term is just the previous cells, the C of T minus one.
How much of this we end up keeping is controlled by the forget gate f of T.
So controls how much of C of T minus one we want to forget.
The second term is the simple answer.
How much of this we end up keeping is controlled by the input gate.
I have T and notice that this has exactly the same equation as the simple aan n unlike the G are you?
Now, note that for the simple answer, we have the activation function f of see, this can be anything,
but usually by default, it's the 10h.
Finally, we have the hidden state h of T, which is just a simple transformation on the cell state
C of T.
In particular, we apply another activation function f of H, which also can be anything but by default,
it's also usually a 10 h.
We then multiply this by the output of T.
So the Applegate controls which values of the cell state we actually pass through to the head and state
of T.
Note that in TensorFlow and Keras, it's actually not possible to choose F, C and F of H individually.
Of course, if you were writing your own LDM, you could customize this any way you wanted.
But for TensorFlow and Keras, this is controlled by the activation argument as usual.
So if you set activation to Rel you, then both of these would be changed to allow you.
So you can see that by building up our knowledge first through simple answers, then she argues, then
storms, we can make something complex like the lithium seem very simple.
It's nothing but three logistic regression ins or neurons to give us the forget gate, the input gate
and the output gate.
Then the cell state is the weighted sum of the previous cell state and the simple ring in weighted by
the forget gate and the input gate.
And remember, we want to offer the cell state the possibility to remember its old state so that we
can learn long term dependencies.
Finally, the hidden state h of T is just a squashed the version of the cells there with the output
gate controlling which values are allowed to pass through.
By the way, I say squashed because most of the time we will use the default 10h activation.
So this is really not a hyper parameter you have to play around with.
Just choose the tarnish by default.
As a side note, if you change the activation, your team unit won't use the GPU, at least in this
version of TensorFlow.
There are some other requirements, too, for the team to be GPU compatible, so you'll want to check
out the TensorFlow documentation for the most up to date list.
In terms of code, how do the GIU and loss team work as promised?
These have the same API as the simple answer.
So if all you want to do is plug in a Geu or LSM in place of the simple aren't in, it's as simple as
changing the name.
In other words, just type in a different name, and all of a sudden you have a more powerful model.
Easy.
Later on, there will be some useful options for these Arnon units that we can make use of.
I want to briefly mention needs to you now so that you know they exist, although we won't be making
use of them right away.
First, recall that for each input X of one x of two, all the way up to X a big T. We will calculate
ahead and state each of one of two all the way up to each of Big T..
Currently, we've seen that all the art in units, the simpering in the GIU and the LSHTM only return
at this final hidden value.
Each of Big T.
But there are scenarios where we would want the other headed states to so that we can calculate the
corresponding output predictions we had of one way out of two and all the way up to why had a big tea.
In order to do that, it's very simple.
All we need to do is pass in the argument return sequences equals true when you do this.
The output of the orange unit will be of shape and the bitey by em because remember the hidden feature
vector is of size m.
And then when you apply a dense layer on top of this, your final output is a shape and by T by K where
K is the number of output nodes.
So you have an output for every time, step, for every sample.
Another interesting argument is the return state argument for the simple Arnon and the Giu.
This is a bit superfluous.
We already know that the output is the head in state.
So when we use the functional API and we grab the return value, this is already the head and state.
So if you set returns, state equals true.
You'll just get back the same thing.
Return twice for the yam.
The cell state is also considered a state, so if you set returns, state equals true for the LSM.
It overturned three things the output the head and state and the cell state, the output and the head
and state are still the same thing, but the cell state is different.
Again, this is kind of random information at this point since we won't be using it immediately, but
it's good to have in the back of your mind that just in case.