In this lecture, I hope to provide you with a very intuitive introduction to recurrent neural networks.
This lecture will focus on the simple rainin, also known as the Elleman Unit.
So let's begin this lecture by thinking about why we might want to use our own ends.
Suppose that our task is to classify words.
The label doesn't really matter, but you could think of it maybe as parts of speech.
For example, whether the word is a noun, a verb and so forth.
Now suppose that we use a regular feed forward and then to solve this task, our CNN simply takes in
a single word and outputs a label whi, which is a prediction for the label.
OK, so nothing special so far.
But what's the problem with our island?
Well, the problem is there are multiple possible answers which depend on context.
For example, if the previous word was behnke and we know that bank has several meanings.
One obvious example of a bank is a financial institution.
If this appeared in your email, it might have a higher chance of being classified as spam.
But we know that the word a bank can mean other things.
For example, a river bank if your friend invites you on a hike along a river bank.
This clearly has a low chance of being spam.
The word bank can also be a verb.
So if you're doing parts of speech tagging, then your tag could be noun or verb, depending on the
context.
Thus, what we are saying is the target is not determined by the input alone.
It also depends on the surrounding context.
Now, recall that we interpret opponents as models that can find hidden features in data, we might
think of these as hidden representations.
For example, if we're looking at faces, one of these units might represent eyes while another might
represent years.
So this is a critical point to understand for this lecture, which is just a review from the previous
sections.
The hidden layers of a neural network output hidden representations of the input.
Now, let's go back to our problem of word classification, suppose that for the entire sequence of
inputs, we apply the same neural network to make a prediction as we normally would.
But now let's turn this into an answer on a recurrent neural network for each neural network instead
of only taking in a single input at the current timestamp.
It also takes in the hidden representation from the previous timestamp.
Note that by doing this, there is a pathway from all the previous inputs to the current hidden representation.
So it not only depends on the past timestamp, but effectively all past time steps through the previous
hidden representation.
So this is called a recurrent neural network.
So why is this called a recurrent neural network?
Well, recognize that a much more compact way of representing this network would be to draw a self loop
that is, the hidden layers output goes back to its input with a time delay of one.
In previous years, I always started my explanation of Arnaz with this diagram, but I found that some
students didn't have the intuition for it.
Possibly this is because diagrams like these are common in engineering, but not in other fields.
So if you are an engineer, then you should find this very intuitive.
But if you've never seen anything like this before?
Take note of how it relates to the previous diagram, which we call an unrolled end.
We call it unrolls because this version is like a rolled up version of each time step compacted into
a single diagram.
So let's discuss more about why this is a recurrent neural network now that we understand the flow of
information that we can start thinking in terms of neurons, as you recall, one of the most important
expressions you learn in machine learning is W Transpose X plus B or wait times input plus bias.
Well, it's going to appear again.
Basically, every arrow you see on this diagram mathematically means take the input multiplied by W,
and I'd be again, remember that you shouldn't think of this as math, but merely a pattern of symbols
that you'll see again and again.
Don't think of it as math, but rather just as a familiar concept.
Now, recognize that the WS and B's are different depending on the role of the arrow.
If it's an arrow going from one hidden representation to another, we'll call the weights and b h.
If it's an arrow going from the input to the head and representation, we'll call the weights W X and
B X.
Note that from this point forward, we will call the hidden representations Hidden States.
This is common terminology when we're dealing with sequences.
OK, so what happens when we try to turn this diagram into math?
Well, suppose that we're given some prior head in state h0.
From this, we can compute the linear transformation, which transpose times zero plus b h.
At this point, we also want to transform the input x one.
From this, we get W X transpose times X1 plus b x as per our diagram.
We then add these two together and apply some activation function, like the value of the tannic h.
This gives us a new head in state h one.
So what happens at the next step?
Well, we do the same thing again.
We transform each one using transpose times, each one a plus b h.
We also transform X two by using a W X transpose times x two plus b x y z.
Again, add these together and apply the activation function.
This gives us h two.
Now, of course, we can write this more compactly by recognizing that all the steps look the same,
we can say each of T is equal to sigma of W.H. transpose times h of T minus one plus b h plus w x transpose
times x of T plus B X. So now it should be clear why these are called the recurrent neural networks.
This equation is a recurrence as a simple example of a recurrence.
Think of the Fibonacci sequence it says X event is equal to X event minus one plus x event minus two.
So it's a similar idea where the current value is computed by applying some transformation on the previous
values.
As usual, it's always important to think about shapes.
So given this recurrence, can we infer what the shapes of our weights and bias factors should be?
Let's start by recalling our convention that we normally use em for the size of the head and vector
and for the size of the input vector.
Since W.H. has to map a size and vector to a size M vector, it must have the size M by M.
Since W X has the map of size d vector two a size m vector.
It must have the size of D by m now as per the and an S. Many students often ask Why do we use D by
M and not my D?
Surely this would be more convenient.
Since then, we don't have to transpose our weight matrices.
However, keep in mind that this is simply the convention when we write things mathematically.
Again, one way to see this is to think of a single value in The Matrix WJ.
If we use the convention input size by output size, then this value always represents the way going
from input ie to output j.
In other words, the input comes before the output, which is more intuitive.
Finally, it should be clear that both B and B X must be vectors of size M.
You should also recognize that there are multiple ways of representing this recurrence.
For one, it seems that having multiple bias terms is redundant.
It would be simpler if we just had one.
So that's one way of simplifying this recurrence.
Another way to simplify this recurrence is by combining H of T minus one and X of T.
Into a single vector and combining W H and W X into a single matrix.
In this case, the combined input vector has the shape d plus m.
Since this needs the map to h of T, which is a vector of size m, the combined weight matrix has the
shape D plus m by M.
And of course, an equivalent way of looking at this is that it's just W X and W h concatenate it into
a single matrix.