In this lecture, we are going to extend our understanding of convolution so that it looks a little
more like what goes on in a neural network.
First, remember that in general, a neural network accepts as input color images.
We know that these are three dimensional objects, height by width, by color.
But for all the examples of convolution, we just looked at both the image and the filter were two dimensional.
So how can we reconcile this discrepancy?
Previously, we conceptualized an image as a square and the filter also as a square, then we would
slide this little square along the big square and do multiply add.
And that's convolution.
So if our image has three dimensions, we just go with the same analogy.
Our color image is a big box and our filter is a little box.
Then we slide this little box along the big box, and at each position we do a multiply add and that's
a convolution.
Here are the details at each position.
We used to some over the height and width of the col because the colonel was 2D, but now since the
colonel is 3D, we're going to sum over the color channel as well.
Note that both the colonel and the image four now have three color channels.
If you recall, that's red, green, blue.
The way that you can think of this is that instead of the filter being a grayscale pattern finder,
it's now a color pattern finder.
Suppose the filter is looking for a red circle, so there's a circle in the red channel.
Now it's only going to match red circles.
It won't match blue circles or green circles.
This is opposed to the black and white filter, which has no color at all.
If it saw a circle, then the pattern would be found.
There's no concept of color in a black and white image.
Ultimately, this picture is still not complete using what we just learned, this is what would happen.
Our input image is a three dimensional object.
Height by width by three.
Our kernel is also a three dimensional object, K by K by three.
The output of this convolution is only two dimensional.
Height minus K plus one with minus K plus one.
And this is because we're summing over all three dimensions.
And so the third dimension kind of gets sorted out now.
There's nothing necessarily wrong with the output being two dimensional.
Although this should offend your sensibilities to a certain extent.
Remember this idea of the uniformity of a neural network and how we have repeating structures.
The feedforward A9 works because you can add any number of layers one dense layer followed by another,
followed by another.
This is OK because the input and the output are of the same type.
The input is a vector and the output is also a vector, which can then be the input into the next layer.
But if a convolutions so far, this doesn't seem to work if the input is three dimensional, but the
output is only two dimensional.
Then we do not have this uniformity.
We can't plug in the same kind of convolution right after the first one since the input is not the same
as the output.
But here's a question to consider.
Remember that the filter can be thought of as a pattern finder, so it's looking for a particular pattern
or, in other words, looking for a particular feature.
But we know that in machine learning, we have multiple features, just like how each hidden unit in
a neural network represents a different feature based on the inputs.
That kind of makes sense.
If we're looking at an image, we would want to identify multiple features in the image.
For example, let's say you're building a facial recognition network.
One filter might be looking for an eye and another filter might be looking for a nose.
So it makes sense, then, that if we want to find multiple features, we should have multiple filters
to find them.
So consider for a second what would happen if you had two filters acting on the same input image called
the input image a call the first filter one call the second filter W2.
The they'll put a vague and vaults with one is be one, and the output of a car involved with W2 is
B to, let's say for simplicity's sake that we're using same motor convolution so that B one and B to
have the same height and width as a so of A is each by W by three, then B one is h by W and B two is
also h by W.
Now we have two outputs B one and be two, but with different features found in each.
What should we do with them?
Well, how about we simply stack them together?
Now, all of a sudden in our output is three dimensional, its height by width by two.
And in fact, we can add any number of filters to find any number of features.
So nose, eyes, lips, eyebrows, chin, so on.
Now, our input looks like our output.
They are both three dimensional.
This means we can have multiple layers of convolution one, followed by the other in a repeating pattern,
just like we do with an.
But of course, just like with the Ann and Dennis there, we want to victories this operation so that
it can be done all in one go rather than doing, say, 10 different convolutions and then stacking them
together.
So here is the formula that represents how to do a complete convolution in a deep neural network so
that both the input and the output can be three dimensional objects.
Now, this is a lot to look at all at once.
So let's try to consider each part one step at a time.
First, let's consider the shape of everything.
We know that a will b h by W by C one, you can think of C one as the number of color channels the image
has.
Then we have the filter w, which has the shape c one bouquet by K by C to note that this is now a four
dimensional tensor.
Then we have the output image of B assuming that we're doing C mode convolution.
The height and with dimensions will be the same each by W, but the third dimension will be C two.
So in our equation for the convolution, we can see that it's pretty much the same as before.
Just with more indices, BJC represents the image pixel value at Row II Column J and Color Channel C.
Note that I use the term color loosely here because it doesn't really represent color anymore.
We can have an arbitrary number of features, so C two can be 10 120000.
Conceptually, however, we're still just doing what we talked about previously, just individual convolutions,
which output two dimensional images and then stacking all those images together to get a three dimensional
object.
So let's just summarize what we learned in this lecture, since that was a bit long.
First, we started with realizing that we only defined convolution for grayscale images and with corresponding
two dimensional filters.
This is, of course, not flexible enough for CNN's, which must accept color images as input.
So then we said, OK, let's just extend that by doing a three dimensional dot product between a color
image and a three dimensional filter with the same number of color channels.
So we're dotting over the color channel.
Next, we realized that this breaks uniformity so we can't stack layers together to build a deep neural
network.
And this is deep learning, after all.
So we kind of need to be able to do that.
And this is because when you involve a 3-D image with a 3-D filter, you still get a 2D image as output.
This is just like when you dot a one vector by another one vector, you get a scalar.
So we extended this by realizing that we don't just want one filter that's going to find a one single
feature.
We need to find multiple features.
Therefore, we need multiple filters, which will output multiple 2-D images, which of course, can
be stacked into a 3D image.
Then we voucherize this operation so that both the input and output are three dimensional.
Now, you might be wondering, is it correct to call the final dimension color that may be the case
for their input into the neural network, which is always hype by with by three, but this breaks down
at every other layer.
For example, if you have hypo with by 10, then the 10 dimensions don't really represent color.
But as we discussed conceptually, we're just finding different features in an image.
So our terminology changes to cover this more general case, we call these feature maps.
This makes sense because you can think of each 2-D image as a map, as in a literal map you can look
at, which is the result of a convolution with some filter, which is looking for some feature.
So this map is telling us where in the original image that this feature can be found.
So just like a map, the final output is just a stack of all these different maps.
One map for each feature.
And so the size of the final dimension is referred to as the number of channels or the number of feature
maps rather than colors.
Finally, I want to talk about how convolution is going to look as a neural network layer.
If you recall, you can think of convolution as just matrix multiplication.
But with shared weights and as you know, in a feedforward and we don't just have matrix multiplication
at each layer.
We do some other things as well.
In particular, we can add a biased term and apply an activation function.
Well, a convolution layer works the same way.
We're going to add a base term and apply an activation function, because remember, this allows us
to find a nonlinear features or nonlinear functions of the input, which are more powerful than simple
linear functions.
Importantly, we have to consider what the shape of the bias term should be, given that the output
of the convolution is a three dimensional image in a dense layer.
If we have a vector of size m, then a biased term is also a vector of size M.
In other words, there is one scalar for each element in the result.
But for convolution, the bias term will not have the same shape as W convulsed with X, which is a
three dimensional image.
Now, you might think, isn't this equation invalid if B does not have the same shape as W can with
X?
Technically, this shouldn't be allowed by the rules of matrix arithmetic.
If we want to add two matrices together, they should have the same shape.
Technically, this is true, but this is conceptual only, and this will actually work in code due to
the rules of broadcast.
So the bias vector B is only one dimensional and it has the size C two.
If W can vote with X as a shape each by W by C two.
In other words, the same bias term applies to every pixel for each feature map.
As a sign, no convolution is generally a commutative operation, so it doesn't really matter if you're
right w star X or X start w.
Although this is not entirely true in the deep learning case, since W is four dimensional annexes three
dimensional.
Nonetheless, this is just conceptual only, so let's pretend that W comes first so that it looks more
like the dense layer equation.
All right, so now that you understand how convolution and a full convolution layer works in a neural
network, let's consider our savings.
Remember that one way to look at convolution is that its parameter sharing sharing parameters means
that we use the same weights in multiple places, reducing the number of total weights we need.
So let's do a quick calculation to determine our savings.
Suppose our input image is of size 32 by 32 by three.
A modest size image.
Now, let's suppose the size of the filter is three by five by five by 64.
So there are 64 output feature maps.
Then our output image will have a shape 28 by 28 by 64.
This is because 32 minus five plus one is 28.
And let's just assume we're using valid mode convolution.
The number of parameters we needed to compute this output is just three times five times five times
64, which is 4800.
Now, let's compare that to the case where we would use a feed for neural network with a flattened image
like we did in the previous section, then our input would be 32 times 32 times three, which is three
thousand seventy two.
Our output again would be a size 28 by 28 by 64, which is fifty thousand one hundred seventy six.
So if we use the dense there, we would have a weight matrix of size three thousand seventy two times
fifty thousand one hundred seventy six, which is approximately 154 million.
Now that's a lot of parameters for just one layer of a neural network.
In fact, compared to using convolution, we have about 32000 times more parameters.
As you can imagine, this would require a huge amount of RAM and a huge amount of compute.
It would take longer and would perform suboptimal because, as we stated earlier, convolution is a
pattern finder.
We use the same filter to find the same pattern in multiple places without shared weights.
Every weight needs to learn the pattern at every location.
The pattern could appear, which is suboptimal.
This would be bad for a generalization.
Using this concept of convolution, just being part of a neural network layer, it's easy to see how
are you going to find these convolution filters through the process of training at this point, we've
come quite a long way since our first introduction to convolution, where we had the perspective that
convolution is just an image modifier.
The examples we use were blurring and edge detection.
We took an image and then convolution modified it in some way.
But now we have a deeper understanding of convolution.
It's in fact a pattern finder and a shared parameter matrix multiplication.
It's just a feature transformer.
In this way, it's just like the neural network layers we'd seen earlier in the course.
And so having this perspective, it becomes pretty obvious how these filters or feature transformers
will be found during the training process.
The answer is that it's all going to happen automatically when we call model that fit.
This is going to end up doing gradient descent on our new weights and biases, which for convolution
just happened to be a different shape than what we had before in the dense layer.