In this lecture, we are going to discuss how we can use convolutional neuron that works for sequences,
specifically text, although the type of CNN we are about to discuss will work just as well for generic
sequences to.
You might be wondering, how can this be if CNN Zafra images?
How is it possible for CNN to also work with sequences?
First, let's recall the basics of convolution.
The idea is this you have an image, which is the big square.
You have a filter, which is the small square, and you're going to slide the filter along each possible
position in the big square.
And at each point, you're going to multiply all the overlapping values and add them together just like
a DOT product.
Now, I know this is obvious, but it's worth taking note of an image has two dimensions, height and
width.
There is also the feature dimension.
But that's not an actual dimension which has correlation.
And recall what I mean by correlation.
If you have a picture of a red car and you find a pixel in the image which is red, probably the neighboring
pixels are also red.
In other words, pixels beside each other likely have similar values.
Now, think of a sequence, unlike an image, a sequence has just one dimension that is not a feature
time.
So it's time instead of space.
But notice one important detail.
We have the same type of correlation and sequences, data which are nearby.
Other data in time are also close in value.
That's why this appears like a smooth curve rather than just noise jumping around with random, uncorrelated
values.
This suggests that convolution might be useful here as well.
Luckily, convolution in one dimension is actually much simpler than a convolution in two dimensions
instead of a big square.
We just have a big line and the filter is a smaller line.
Then we slide the small line across every position in the big line.
Multiply all the overlapping values and add them together.
Let's do a simple example for the sequence data we have.
One, two, three, two, one.
And for the filter, we have simply plus one, minus one as an exercise.
You might want to try to figure out the answer first before we move on to the solution.
OK, so here's what we get at the first position we have.
One times one plus two times minus one, which is minus one.
At the second position, we get two times one plus three times minus one, which is minus one.
At the third position, we get three times one plus two times minus one, which is one.
And at the fourth and final position, we get two times one plus one times minus one, which is one.
So essentially exactly what we would expect.
As with images, it's possible to express this operation as an equation.
And remember that in deep learning, while we call this convolution, pure mathematicians and statisticians
would call this cross correlation.
So just keep in mind the sign is reversed as an exercise.
You might want to try and confirm to yourself that this equation does, in fact, implement the operation
we just performed in the previous example.
And remember that another thing we can do is add on features.
So imagine X as a T by D array where T is the number of times, steps and D the number of input features.
Then imagine the output as another two dimensional array of size T by M, where T is the number of times
steps and M is the number of output features.
Then W must be a three dimensional read T by D by M.
And then we would have the equation that we see here.
This is just like convolution with images for a two dimensional convolution.
We have two spatial dimensions, plus one dimension for the input features plus one dimension for the
outer features.
So that's four dimensions in total for a one dimensional convolution.
We have one time dimension plus one dimension for the input features plus one dimension for the output
features.
So that's three dimensions in total.
As usual, while convolution might seem like an abstract concept.
Remember that there are multiple convenient perspectives on it that make it seem like things we are
already familiar with.
So one perspective is that it's just matrix multiplication.
Just like a regular feed, flawed neural network.
Except that we have shared weights in order to take advantage of the special structure and correlation
and the data.
Another intuitive perspective is that it's just a sliding dye product and the dye product is just a
correlation finder.
Correlation is just another name for a pattern matching or similarity.
So really what we are doing is asking, is this part of the sequence similar to my filter?
And thus the filter becomes a pattern matcher or a pattern finder.
All these concepts that you learn about convolution before still apply here.
All right, so now that you know how Convolution works with sequences, how do we apply these the text?
Well, luckily when we use them beddings, that already gives us exactly what we need for a one dimensional
convolution.
We need an input, which is a TBD sequence, where to use a number of timestamps, and this is the number
of features.
And of course, this is exactly what we get after we pass our sentence through and embedding layer.
We go from a length to sequence of words to a length t sequence of integers to a length T sequence of
D length vectors.
Since we have T vectors each of length D this makes up a T by the Matrix.
When you stack them all together and thus we have exactly what we need to build a CNN four text.
So here's how it all looks in code.
First, we have our input.
That's a lengthy sequence of word indexes.
Then we have an embedding layer.
The output of that is a TBD sequence of word vectors.
Then we have a one de convolution.
This is just the kind of one declasse.
Then we follow the same typical CNN architecture where we have convolution followed by pooling and so
forth.
So the same pattern applies here that we had four images.
Generally speaking, the data shrinks in the time dimension, but grows in the feature dimension.
So that's why you see the number of feature maps getting larger and larger.
Once we've done that, then we can do a flatten or we can do a global max pooling, which will give
us a single vector of size M three.
Finally, we pass this through one or more dense layers to get a single scalar, assuming we are doing
binary classification.
As you can see, this is no different from a CNN meant for images, except that all the convolutions
and puling are one dimensional instead of two dimensional.
There is one small caveat to what we just discussed.
But it's not obvious until we tried to implement the forward function inside the constructor.
There's no real problem because all we're doing is instantiating some objects.
But remember that in Pae Torch, the general convention is that for convolution features come first.
We saw this earlier in the context of two dimensional convolution on images.
Pae Torch Convolution works on images of size and by C, by H, by W, which means that color comes
before the spatial dimensions.
In other words, a feature comes first.
However, in Tensor flow, open, c.v and other libraries will use the convention.
And by H.
By W, by C.
Which means that the feature dimension comes last.
Tortue vision in some sense accepts that this is the convention and makes its interface such that you
can work with N by H, by W, by C all throughout your code and never have to worry about the fact that
in pi tausche things are backwards.
This is because you don't ever interact with the data explicitly your vision.
Does all that work for you?
But now we have this mixture of conventions.
The output of an embedding is N by T.
By D.
Where T is the sequence length and D the feature dimension.
So the feature comes last.
When it's the output of an embedding.
But the convoy when B function, just like the continuity function, expects the features to be first.
In other words, it expects the features to come before the time dimension.
So it wants to see end by D, by T and not and by T, by D.
Well, we end up having to do is per muting the data right after the embedding and before the first
convolution.
So after the embedding its features last, but before the first convolution, we reshape it to features
first.
Then after all of our convolutions and pullings have been done, we permute the data back from features
first.
It features last.
At that point, we can pass the data through the usual dense layers that make up the rest of the CNN.
So here it is in action, since there are many layers in a CNN.
I've split this up to make it more readable.
Here we take in the input and pass it through and embedding layer.
Then we do a series of convolutions and pullings.
I'm not going to show all the convolutions and pullings since we just repeat the same pattern over and
over.
But we can reason about the shapes.
So first, the input X is an end by T array of integers.
After the embedding it becomes N by T by D word vectors.
Next we permute the dimensions.
So it becomes N by D.
By T.
After the first convolution, we change the number of features.
So it becomes and by m by T the real you which comes next, doesn't change the shape of the data.
After puling, which changes the length of a sequence we get entered by M by T two.
You can think of T two as T divided by two or two divided by three depending on the pulling parameters.
Next we go through the second convolution and so on and so forth.
So let's pretend that we had a three convolutions.
So after the final convolution, our data has the shape.
And by M three by two, three at this point, we need the features to be last again.
So we call permute again and we get in by T three by M3 three after doing a global max pool along the
time dimension we get.
And by M three, note that because we can specify the axis to perform the global max pool on, we didn't
necessarily have to permit the data after the final convolution.
You could just choose a different axis.
Although I think this makes it more clear what's going on.
Lastly, we passed the data through a final dense layer and as usual, we get back an array of size
and by K!