In this lecture, I hope to provide you with a very intuitive introduction to recurrent neural networks.
This lecture will focus on the simple RNA, also known as the Alman unit.
So let's begin this lecture by thinking about why we might want to use our brains.
Suppose that our task is to classify words.
The label doesn't really matter, but you could think of it maybe as parts of speech, for example,
whether the word is a noun, a verb and so forth.
Now suppose that we use a regular feed forward and then to solve this task, our anend simply takes
in a single word and outputs a label whi which is a prediction for the label.
OK, so nothing special so far.
But what's the problem with our and.
Well, the problem is there are multiple possible answers which depend on context.
For example, if the previous word was bank and we know that bank has several meanings, one obvious
example of a bank is a financial institution.
If this appeared in your e-mail, it might have a higher chance of being classified as spam.
But we know that the word bank can mean other things.
For example, a riverbank if your friend invites you on a hike along a riverbank.
This clearly has a low chance of being spam.
The word bank can also be a verb.
So if you're doing parts of speech tagging, then your tag could be noun or verb, depending on the
context.
Thus, what we are saying is the target is not determined by the input alone.
It also depends on the surrounding context.
Now, recall that we interpret Anan's as models that can find hidden features in data, we might think
of these as hidden representations.
For example, if we're looking at faces, one of these units might represent eyes, while another might
represent ears.
So this is a critical point to understand for this lecture, which is just a review from the previous
sections, the hidden layers of a neural network output, hidden representations of the input.
Now, let's go back to our problem of word classification.
Suppose that for the entire sequence of inputs, we apply the same neural network to make a prediction
as we normally would.
But now let's turn this into an hour and then a recurrent neural network for each neural network instead
of only taking in a single input at the current time step.
It also takes in the hidden representation from the previous timestep.
Note that by doing this, there is a pathway from all the previous inputs to the current hidden representation.
So it not only depends on the past timestep, but effectively all pass time steps through the previous
and representation.
So this is called a recurrent neural network.
So why is this called a recurrent neural network?
Well, recognize that a much more compact way of representing this network would be to draw a self loop.
That is the hidden layers.
Output goes back to its input with a time delay of one in previous years.
I always started my explanation of Arnez with this diagram, but I found that some students didn't have
the intuition for it.
Possibly this is because diagrams like these are common in engineering, but not in other fields.
So if you are an engineer, then you should find this very intuitive.
But if you've never seen anything like this before, take note of how it relates to the previous diagram,
which we call an unrolled or an end.
We call it unrolls because this version is like a rolled up version of each timestep compacted into
a single diagram.
So let's discuss more about why this is a recurrent neural network, now that we understand the flow
of information, we can start thinking in terms of neurons.
As you recall, one of the most important expressions you'll learn in machine learning is W, transpose
X plus B or A wait times input plus bias.
Well, it's going to appear again.
Basically, every arrow you see on this diagram mathematically means take the input multiplied by W
and I'd be again, remember that you shouldn't think of this as math, but merely a pattern of symbols
that you'll see again and again.
Don't think of it as math, but rather just as a familiar concept.
Now recognize that the WS and B's are different depending on the role of the arrow.
If it's an arrow going from one hitting representation to another, we'll call the weights and b h if
it's an arrow going from the input to the head and representation, we'll call the weights and backs.
Note that from this point forward we will call the head and representations head and states.
This is common terminology when we're dealing with sequences.
OK, so what happens when we try to turn this diagram into math?
Well, suppose that we're given some prior head and state H0 from this.
We can compute the linear transformation, transpose times, H0 plus B.H. at this point.
We also want to transform the input X1 from this.
We get X Transpose Times X1 plus B X.
As for our diagram, we then add these two together and apply some activation function like the value
of the Tanach.
This gives us a new head and state each one.
So what happens at the next step?
Well, we do the same thing again, we transform each one using transpose times, each one A plus B,
H, we also transform X two by using X Transpose times X two plus B, X, we again add these together
and apply the activation function.
This gives us two.
Now, of course, we can write this more compactly by recognizing that all the steps look the same,
we can say each of T is equal to sigma of transpose times, each of T minus one plus B plus X transpose
times X of T plus B, X.
So now it should be clear why these are called recurrent neural networks.
This equation is a recurrence as a simple example of a recurrence.
Think of the Fibonacci sequence.
It says event is equal to X, even minus one plus X event minus two.
So it's a similar idea where the current value is computed by applying some transformation on the previous
values.
As usual, it's always important to think about shapes, so given this recurrence, can we infer what
the shapes of our weights and bias factors should be?
Let's start by recalling our convention that we normally use em for the size of the head and vector
and for the size of the input vector.
Since W.H has to map a size and vector to a size and vector, it must have the size MBM.
Since W X has the MAPA sized vector to a size M vector, it must have the size of D by M now as per
the Annan, many students often ask why do we use DVM and not Amedee?
Surely this would be more convenient.
Since then, we don't have to transpose our weight matrices.
However, keep in mind that this is simply the convention when we write things mathematically.
Again, one way to see this is to think of a single value in the Matrix YJ If we use the convention
input size by output size, then this value always represents the weight going from input ie to output
j.
In other words, the input comes before the output, which is more intuitive.
Finally, it should be clear that both B.H. and the Becks must be vectors of size.
You should also recognize that there are multiple ways of representing this recurrence, for one, it
seems that having multiple byas terms is redundant.
It would be simpler if we just had one.
So that's one way of simplifying this recurrence.
Another way to simplify this recurrence is by combining H of T minus one and X of T into a single vector
and combining X into a single matrix.
In this case, the combined input vector has the shape D plus M since this needs to map to each of T,
which is a vector of size M, the combined weight matrix has the shape D plus M by M and of course an
equivalent way of looking at this is that it's just W, X and y h concatenate it into a single matrix.