In this lecture we are going to begin discussing recurrent neural networks.
Let's first think about why an end would be useful first.
As usual we are going to start with our dumb as possible approach.
What is the simplest thing we could do.
Let's think back to how we approach the image classification.
Before we learned about CNS which are specialized for looking at images we used an ETS.
We were able to do this because we took an image and we just flattened it into a feature vector which
is the only type of data you can pass into an ANZ.
Recall that this goes along with our model.
All data is the same.
It doesn't matter if your feature vector is a flattened image or some data you collected in a survey.
The ANZ doesn't care
so let's consider whether we can take the same approach now.
In fact we already did when we looked at our time series forecasting model.
We just said pretend t is D and pass that end by Team matrix into a linear regression model.
The model has no idea what you're passing in as a time series.
It's just doing the same algorithm it's always done.
But here's the question what if your D is not one.
In other words you have a multi-dimensional time series recall that one example of this is if you work
at neural link and you've implanted multiple electrodes into your brain.
Now you have multiple recordings at every time step making it a multi-dimensional time series
Well here's an idea that requires pretty much no imagination at all.
Why not flatten this data to let's take a TBD matrix and turn it into a t times D sized vector.
Let's say you have five electrodes in your head and you make a recording with sequence length one hundred
so one hundred times steps.
Then if you flatten this you would just get a vector of size 500
in fact this is even easier to visualize than a flattened image.
Here we just have five different times three signals.
Now all we're going to do is concatenate them together and say it's one big vector.
Easy peasy
all right but we already know that this approach is possible and we've pretty much done it in code already.
If you're confused what I mean by that it would be pretty much the same as our M.A. script which at
this point is pretty simple compared to what we've done so far so let's go to the next step and think
about why that might not be a good idea.
The first problem is the same as with CNN as if you were to do a full matrix multiplication.
It would just take up way too much space if d equals one hundred and T equals ten thousand which by
the way is not unrealistic.
Then all of a sudden you're flat in a feature vector is of size 1 million.
You probably don't want a one million size feature vector going into your and n
as with CNN s aren't and take advantage of the special structure of the data and CNN is the most general.
It just connects every input to every feature every feature in other words is directly composed of every
single input.
We know that for images this is not really a great idea because we're actually looking for very tiny
patterns relative to the full image.
The question is In what way can we exploit the structure of a sequence
here's the idea we can take inspiration from forecasting.
We know that to predict the next value past values are useful but what if we apply this to the hidden
feature vectors.
So let's take an ANZ where the hidden feature vector is calculated from the input vector.
The output is then calculated from the head in vector
now to make an aunt in all we need to do is make the head and vector loop back to itself.
In other words the hidden vector depends not only on the input but also on its own previous value.
This is just like our very basic linear regression forecasting model where the prediction was just a
linear function of past values.
Now we are saying the hidden state is a nonlinear function of past values specifically that the nonlinear
function is a neuron.
As a side note the loop back implicitly assumes that there's a time delay of one step
all right so how do we calculate the output of an Ana.
Well we use the calculation you see here as you can see there's only one new thing the head and state
age of T now depends on the hidden state at the previous time step as well.
Also x y and H are index by time but that should be a given.
Some other assumptions we made here are that we're using the sigmoid activation for both the hidden
and output layers.
This is of course not necessary but we need to write something down so that's what I decided to write
down.
As usual output activation is dependent on the tasks being done.
So a sigmoid for binary classification.
Nothing for regression in a soft Max for multi class classification.
Another assumption is that this neural network only has one hidden layer.
In fact with orange ends this is pretty often the case.
You could stack up more hidden layers but I don't see that too often.
As usual you'll want to test that out on your particular dataset.
If you think it might be a good idea
it's helpful to name these weight matrices so we don't get them confused later on.
Luckily it's pretty intuitive.
The reason we call w x H W H is because the input is X and the output is H.
So we call this the input to head and wait.
The reason we call w h h w h h is because the input is the previous H and the output is the next H.
So we call this the head into head hidden way B of h we call the hidden bias and finally w o and below
are the output weight and output bias.
The important thing there was to differentiate between w x h in w h h since they are both weights belonging
to the recurrent layer.
As a side note this layer itself has several names the most common name is just the simple recurrent
unit but since this has been studied in the past it's also associated with a person.
And so it is also known as the mean unit.
All right so for most people these equations are pretty intuitive but for some it's not.
So let's just be super clear about how we were going to calculate the output prediction given a sequence
of inputs.
Actually this will help us uncover some hidden details that we need to consider that are not obvious
at first.
First we assume that we are given a sequence of input vectors x 1 all the way up to X Big T each individual
X is of course a vector of size D.
So this entire sequence would be stored in code as a matrix of size bigger t by D.
In math however we assume that we are working with individual vectors.
So let's start with x 1 from x 1.
We can calculate H1 but wait.
Now you see one of these hidden details that you probably didn't think about H1 depends on h 0.
But what is h zero.
In fact we refer to this as the initial hidden state typically we just set this to an array of zeroes
but it can also be a learned parameter.
In other words you can learn this vector using gradient descent in PI talk.
It's possible to make the initial state a learnable parameter but it's simpler not to.
And in fact quite conventional.
So in this course that's the approach we're going to take.
All right.
So now we have each one from this we can calculate y hat one using the usual neuron formula.
Next we consider the input vector x 2.
From this we can calculate H2 H2 depends on x2 and H1.
We just calculated each one.
So that's not a problem.
From this we can calculate y had two
now we can repeat this process to calculate age 3 and why have 3 age 4 and why have 4 and so on all
the way up to age back T.
And why have Big T
Okay great.
That all makes sense by the way.
If you found that a little confusing don't worry because we are also going to do this calculation in
code as well.
Just as a little extra practice to make sure you understand how it works.
But here's something you probably noticed which is strange.
Why do we have a y hat for every time step if we are doing something like forecasting.
We only want to predict the very next value if we are predicting something like the sentiment of a tweet.
We only have one answer.
Positive sentiment or negative sentiment.
So what is the purpose of having a Y have for each timestamp.
Each of the Y hats depends only on the x's up to that point.
The answer is that for these types of problems which I just described all the y hats except for the
final y and are ignored.
So why hat of Big T gives us the final prediction and all the previous y hats are discarded.
You can think of them as like temporary variables
at the same time.
There are cases where we would want to keep all the y hats at each time.
One such example is neural machine translation in a neural machine translation.
Both your input and your output are sentences.
They are just sentences in different languages.
In this case both the input and the target are sequences and therefore you need to capture the predictions
at each time point
considering neural machine translation.
Let's think about how we can reason about what an Arnon is predicting in probabilistic terms.
If you recall when we look at an ends and hence this also applies to CNN is that the output prediction
after applying the soft max function is a probability distribution for a regular and N or a CNN.
It's the probability that Y equals K given x x is the input vector or in the case of CNN is an image
and y is the label for normal machine translation.
This is also classification.
We are trying to predict a category which is a word in the target language.
So the question for Arnold's is if we are doing classification we will have the probability of Y of
T equals K given something.
But what is this something
one picture that helps us to understand this is the unrolled Arnon.
This is like a block diagram of all of our calculations and their dependencies.
First we can see that each one depends on h 0 and x 1 y had one.
It depends on each one and therefore it indirectly depends on X1 H2.
Depends on H1 and X to Y had to.
Depends on H2.
Thus y had to indirectly depends on X1 and X2 similarly y had 3.
Depends on X1 X2 and X3.
Why have 4 depends on X1 X2 X3 and x 4.
Finally y 5.
Depends on all the X's x1 x 2 x 3 x 4 and x 5.
In general we can say that Y had a big T.
Depends on every X in the input sequence.
It should be clear from this picture that each y of T overall depends only on the current and past values
of x.
So what does this mean in terms of the probability distribution that the Arnon is modelling the probability
of Y when equals K given X1 is the first distribution.
But now notice that my have to depends on both x 2 and each one which in turn depends on x 1.
So we write the probability of Y of 2 equals K given x 1 and x 2 Next we have that the probability why
three equals K given x 1 x 2 and x 3.
And this pattern continues all the way up to the probability of Y a big T was K given x1 x 2 x 3 all
the way up to expert T.
Now this is not that important in and of itself.
You're not really going to be manipulating these possibilities or doing anything mathematically but
it's interesting.
If you've ever studied Markoff models the idea with the markup model is that they make the mark of assumption.
What is the mark of assumption.
It's easiest to understand with words the Markov assumption says that the probability of the next word
depends only on the current word in the sentence but not on any previous words.
You might think that sounds very unrealistic.
For example if the current word in a sentence is the how can you possibly predict the next word in a
sentence the next where it could be any number of words.
So Mark our models are pretty weak if you use them that way
now consider an R and then for predicting the next word in a sentence.
So we're modeling the probability of x of T plus 1 the word at time T plus 1 then our output distribution
is the probability of X at T plus 1 given X of one x of 2 all the way up to x of T.
As you can see the Arnon accounts for all the previous words in a sentence rather than just the most
recent word.
This is one reason why our own ends are so powerful for modeling sequences
as a final thought exercise for helping you understand or an ends.
Let's write some pseudocode that would give us the Y has at each time.
First we have to consider what we're given.
We're given the initial state a 0 and the weights of the R N so we have w x H which maps from X to h
we have w h h which maps from the previous H to the current H.
We have B of H the bias term for the recurrent hidden layer we have w o the output way and below the
output bias and of course we're also given a single input sequence X which has the shape TBD.
It represents t sequential vectors of size D.
Let's assume a 10 H activation and a soft Max at the output
okay.
So to calculate the output predictions we're going to start by initializing y had the output as an empty
list.
Then we're going to set H last the previous value of H T H zero then we're going to live through each
timestamp from one up to big T
inside the loop.
We're going to calculate h of T using the recurrent neuron equation we saw earlier.
Then we can use h of t to calculate y hat at time T.
As usual by the way this is just the dense layer next We append white have to the list of predictions
why finally and this step is important not to forget we assign h of t to H last so that H last always
represents the last value of H of t when we do our calculation for the next stage of T
at this point we are going to briefly go back to the biological inspiration for a neuron that works.
Recall earlier when we talked about how neurons are organized in the brain we can imagine that there
is no reason for the neurons in your brain to all go in one direction.
In fact recurrent circuits in the brain have been well studied in biology.
The idea that neurons can connect in a loop goes back decades.
And personally I find this topic very interesting.
One such example is the hot field network one of the methods used to train hot field networks is called
heavy and learning invented by Donald head.
He inspired the famous rule neurons that fire together wire together and neurons that fire out of sync
fails a link.
It's actually quite unfortunate that most people have a very narrowly focused view on deep learning.
It would be very interesting to see if we applied modern technology to these old ideas what we could
discover.
The last thing we are going to do in this lecture now that you understand our own ends is to ask the
question what are our savings.
If you recall we did this example with CNN is also the idea was because CNN is take advantage of the
structure of images we can use shared awaits the same idea applies to art ends with our own ends.
We apply the same weight matrix to each input vector x of tea and we apply the same headings ahead and
wait for each calculation of each of T.
Thus it's useful to compare how much we would save if we use that aren't in instead of banana N N with
a flattened time series.
So let's do a simple calculation for an N for simplicity's sake.
We won't consider bias terms
so let's say that t the sequence length is one hundred and b the feature vector size is 10.
Let's say that the head and vector size M is 15.
Then if we flatten this we would have a 1000 length input vector
correspondingly.
We would also have t hat and states.
So the hint and size in the end would be t times M which is fifteen hundred.
Finally let's suppose we are just doing binary classification so the number of output nodes is just
1 in this scenario.
Our inputs ahead and way it would be of size one thousand times fifteen hundred which is one point five
million are hit into Apple way would be of size fifteen hundred by one which is just fifteen hundred.
Therefore in total we have approximately one point five million weights
now.
Compare this to the simple RNA.
Remember we are calculating the size of W x H W H H and W O W x H.
Since it goes from inputs ahead in must be of size D by M which is 10 times 15 which is 150 w h h since
it goes from head into head n must be of size m by m which is 15 times 15 which is to twenty five.
Finally we have w o which goes from head into output which must be of size m by 1 which is just 15 so
in total we have 150 plus to twenty five plus 50 which is 390.
So if we take an N N which has one point five million one thousand five hundred weights and divide that
by 390 we find that N N N has three thousand eight hundred fifty times more parameters which is a huge
amount of savings.
Thus we learn again that by taking advantage of the structure of specialized data we can create much
simpler and more compact models compared to full and NS.