In this lecture, we are going to discuss how we can use convolutional neural networks for sequences,
specifically text, although the type of CNN we are about to discuss will work just as well for generic
sequences too.
You might be wondering how can this be?
If CNN's are for images, how is it possible for CNN's to also work with sequences?
First, let's recall the basics of convolution.
The idea is this you have an image which is the big square, you have a filter, which is the small
square, and you're going to slide the filter along each possible position in the big square.
And at each point, you're going to multiply all the overlapping values and add them together just like
a dot product.
Now I know this is obvious, but it's worth taking note of.
An image has two dimensions height and width.
There is also the feature dimension, but that's not an actual dimension which has correlation.
And recall what I mean by correlation.
If you have a picture of a red car and you find a pixel in the image, which is red, probably the neighboring
pixels are also red.
In other words, pixels beside each other likely have similar values.
Now, think of a sequence, unlike an image, a sequence has just one dimension that is not a feature
time, so it's time instead of space.
But notice one important detail.
We have the same type of correlation and sequences data which are nearby.
Other data in time are also close in a value.
That's why this appears like a smooth curve, rather than just noise jumping around with random uncorrelated
values.
This suggests that convolution might be useful here as well.
Luckily, convolution in one dimension is actually much simpler than convolution in two dimensions instead
of a big square.
We just have a big line and the filter is the smaller line.
Then we slide the small line across every position in the big line.
Multiply all the overlapping values and add them together.
Let's do a simple example for the sequence data, we have one two three two one.
And for the filter, we have simply plus one minus one.
As an exercise, you might want to try to figure out the answer first before we move on to the solution.
OK, so here's what we get at the first position we have.
One times, one plus two times minus one, which is minus one.
At the second position, we get two times one plus three times minus one, which is minus one.
At the third position, we get three times one plus two times minus one, which is one.
And at the fourth and final position, we get two times one plus one times minus one, which is one.
So essentially exactly what we would expect.
As with images, it's possible to express this operation as an equation.
And remember that in deep learning, while we call this convolution, pure mathematicians and statisticians
would call this cross-correlation.
So just keep in mind that the sign is reversed as an exercise, you might want to try and confirm to
yourself that this equation does in fact implement the operation we just performed in the previous example.
And remember that another thing we can do is add on features.
So imagine X as a TBD array where T is the number of time steps and D is the number of input features.
Then imagine the output as another two-dimensional array of size T by M where T is the number of time
steps and M is the number of output features, then w must be a three dimensional array t by d by M
and then we would have the equation that we see here.
This is just like convolution with images for a two dimensional convolution, we have two spatial dimensions,
plus one dimension for the input features, plus one dimension for the output features.
So that's four dimensions in total for a one dimensional convolution.
We have one time dimension, plus one dimension for the input features, plus one dimension for the
output features.
So that's three dimensions in total.
As usual, while convolution might seem like an abstract concept, remember that there are multiple
convenient perspectives on it that make it seem like things we are already familiar with.
So one perspective is that it's just matrix multiplication, just like a regular, feedforward neural
network, except that we have shared weights in order to take advantage of the special structure and
correlation in the data.
Another intuitive perspective is that it's just a sliding DI product, and the DOT product is just a
correlation finder.
Correlation is just another name for a pattern matching or similarity.
So really, what we are doing is asking, is this part of the sequence similar to my filter and thus
the filter becomes a pattern matcher or a pattern finder?
All of these concepts that you learn about convolution before still apply here.
All right, so now that you know how convolution works with sequences, how do we apply these to text?
Well, luckily when we use embeddings, that already gives us exactly what we need for one dimensional
convolution.
We need an input, which is a T by the sequence where to use a number of timestamps and these the number
of features.
And of course, this is exactly what we get after we pass our sentence through and embedding layer.
We go from a lengthy sequence of words to a lengthy sequence of integers to a length T sequence of length
vectors.
Since we have two vectors each of length d, this makes up a T by the matrix when you stack them all
together, and thus we have exactly what we need to build a CNN for text.
So here's how it all looks in code.
First, we have our input.
That's a lengthy sequence of word indexes.
Then we have an embedding layer.
The output of that is a T by the sequence of word vectors.
Then we have a one deconvolution.
This is just the kind of one class.
Then we follow the same typical CNN architecture where we have convolution two, followed by pooling
and so forth.
So the same pattern applies here that we had for images.
Generally speaking, the data shrinks in the time dimension, but grows in the feature dimension.
So that's why you see the number of feature maps getting larger and larger.
Once we've done that, then we can do a flattening or we can do a global max pooling, which will give
us a single vector of size M3.
Finally, we pass this through one or more dense layers to get a single scalar.
Assuming we are doing binary classification.
As you can see, this is no different from a CNN man for images, except that all the convolutions and
pooling are one dimensional instead of two dimensional.