OK, so we've just learned about the simple answer and also called the Elleman units.
Now the next obvious question is how do we use this to solve actual problems at this point?
I'm going to introduce you to two kinds of problems we can solve.
One is called the many to one task.
That means you have a whole sequence of inputs, but only one output.
Some examples of this are spam detection and sentiment analysis.
So for spam, we're going to take in a whole sequence of words, but the target is just a single class,
either spam or not spam.
For sentiment analysis, we have the same idea.
Our input is a whole sequence of words like a movie review.
But the target is just whether or not it was positive or negative.
The second kind of task is called a many to many task.
This is when we have a sequence of inputs and the sequence of outputs.
Some examples of this are parts of speech tagging and anomaly detection in a time series.
So as you recall, parts of speech are labels like noun, verb and so forth.
Clearly, these need to be applied to each word, and thus every word in the input will have a corresponding
target for anomaly detection.
You can imagine that any point in the time series could be classified as anomalous.
For example, you might want to monitor CPU usage on a cluster of machines every point in the time series
would be labeled as anomalous or not.
OK, so what's the point of discussing these different kinds of problems?
Well, this will determine the architecture of the answer.
Conceptually, our neural network will ends with a final dense layer, as it always does.
As usual, it will have a final activation, which is appropriate for the task at hand.
However, recognize that there are multiple ways to connect this final dance layer.
As you recall the art and outputs ahead and state for every time step.
We have each H2 all the way up to HPT.
The question is what do we do with all these hidden states?
The answer depends on which kind of task we are doing.
If we are doing a many to one task, then we only keep the final hidden state, which contains all the
information from the entire time series.
We then pass this through a final dense layer to get a single prediction.
If we are doing a many to many task, then we keep all the hidden states, each of which contain only
the information up to that point.
In this case, we pass every hidden state through the final dense layer to get big teams, separate
predictions, one for each timestamp, and note that the same dense layer is applied to all timestamps.
Just as the same simple Arnon is applied to all timestamps of the input time series.
This is another example of the concept of shared weights, which you may have seen when you studied
CNN's.
Now, there's one more option to consider in the many to one case, this will bring us back yet again
to the concept of shapes.
So hopefully you're beginning to see why thinking about shapes is such an important thing.
So if you studied scenes, this will make a lot of sense.
But if not, that's OK, too.
Just try to think of the intuition.
So suppose we're looking at one Z convolutions.
After doing a series of convolutions and pooling, you'll have a feature race time series of size T
by M where M represents the number of feature maps.
But note something interesting, which is that earnings give us the exact same shape.
After going through an art end with hidden output size M, we will have a T by M sequence.
In this case, we say that we have T hit state vectors each of size M.
So in both cases, the output sizes T by M, but just because they have a different name does not mean
they are different things.
They are just different perspectives on the same kind of data.
One perspective is convolutions, while the other perspective is Arnett's.
Now again, if you haven't seen convolutions before, please don't worry.
Now, as you recall, before passing our data through the final dense layers, we need to obtain a single
flat feature vector.
One way of obtaining such a feature vector is to use global max pooling.
What this does is it takes the maximum value over time, such that you end up with different features.
Put simply, we're getting rid of the time dimension.
It makes sense to pick the maximum, since we use that as a proxy for which value matters the most.
Intuitively, you can think of this in terms of sentiment analysis.
Suppose that we're looking at some movie review and the word terrible appears in the review, but not
necessarily at the end of the sentence due to the vanishing gradient problem, the aunt and might not
be able to recognise the word terrible if it appears too far away from the end.
However, by taking the maximum, we can look at all the hidden values from every time step, which
lets us see more clearly which words in the sentence matter most for predicting the target.
So for the many, to many case, let's consider what the shape of our output would be.
Suppose that we have an input sequence of sheep TBD.
After the orange layer, we will have a head and state vector sequence of sheep t by M.
That is, every time step gets its own head and state vector.
After passing each of these head and state vectors through one or more final dense layers, each with
output shape k, we will have a sequence of sheep t by K. So imagine that our task is to predict parts
of speech of which there are eight kinds.
In this case, K would be eight.
OK, so you can see how this is analogous to and ends with an ends, we have an input vector of size
d, a hidden vector of size M and an output vector of size K with many too many Arnaz.
We have an input of size TBD.
A hidden sequence of size T by M ends an output sequence of size T by K.
Now, one question students always ask is, can you stack multiple aren't and layers?
And of course, this is deep learning, so the answer is yes.
Remember that neural networks are essentially repeating structures if the output of one random layer
is T by one.
That's just another multivariate time series.
We can pass as input through another Arnold and layer.
From this, we'll get another head and state sequence of size T by M2.
Of course, this is just another multivariate time series which we can pass through yet another Arnold
layer ad infinitum.
Now, whether or not this actually helps remains to be seen.
As always, it's just a matter of testing it on your data set to see how it performs.
Here's another way of looking at a neural network with multiple Arnon layers.
Now this is a good time to mention that a pretty common mistake amongst beginners is that they confuse
T, which is the sequence length with M, which is the number of hidden units.
Basically, it's important to remember that these neural networks are much larger than Aon's, so we
can't draw them with the same granularity previously when you saw a circle in a diagram for an A9.
You could think of it like a single number.
But for these range diagrams.
Notice that each circle does not represent a number, but a whole are in a unit which outputs a vector.
Another point to notice is that although it's possible to make it so that each Arnon layer has a different
number of hidden units, it would be unusual to do so.
In other words, it's typically the case that M1 is equal to M2.
One important fact I want to mention is that the architecture you learned about in this lecture apply
to all kinds of Arnon units.
At this point, we've studied the aluminum, then later on we will study that grew and the lithium.
However, although the units themselves are different, the way they are incorporated into our own ends
remains the same.
So when it comes to using R and ends with TensorFlow, although what you are using might be more complex,
the coach's demands the changing what kind of objects you use.
In other words, the interface to all these are in a unit is the same.
OK, so to summarize this lecture, we've just learned how to build art and architectures for two different
tasks.
The first kind of task is the many to one task.
The second kind of task is the many to many task.
In the many to one case, we learned two different options.
The first option is to simply take the final hit and state vector each of Big T and pass this through
the final dense layers.
The second option is to use global max pooling instead.
For many too many tasks, we have to keep all big T hidden state vectors.
We pass each of these through the same final dense layers, and our output has the shape t by K..
Work is the number of outputs for each target.
So for example, if you're doing spam or not spam, then K would just be one or two.