Now that you understand how exactly a convolution layer works including the bias term and activation
function we can now consider the architecture of a convolution or neural network and why it's that way.
So as a little bit of a history lesson modern CNN is essentially all originated from the same model.
The Lean In this is named after Yan LA -- one of the original Deep Learning pioneers along with Jeffrey
Hinton and Joshua NGO.
What we're going to do in this lecture is just start with what the architecture is and then go through
it piece by piece so that you understand why it is the way it is.
Okay so let's look at a typical CNN.
A typical CNN has two stages.
The first stage is a series of convolution of layers.
Importantly these convolution of layers are usually followed by pooling layers which is another type
of layer we'll discuss shortly.
The second stage is a series of dense layers also called a fully connected layers.
These are just a regular feet for a neural network.
As you saw in the previous section.
If you recall I said in the previous section that neural networks have this hierarchical structure.
Each layer is always looking at features that were found in the previous layer so you can even think
of the first stage as just another feature transformer.
It's a feature transformer that specifically works on images and finds IMAGE FEATURES.
Then once it's found those image features it passes those into a neuron that work and then the neural
network does what it does best nonlinear classification or regression
so let's have a closer look at the convolution of stage.
Now I just added a new layer without really explaining it.
The pooling layer.
Why would we want to do that at a high level.
Pooling as a down sampling operation what is down sampling.
That's when you make a smaller image out of a bigger image.
So imagine your images one hundred by one.
After pooling.
If the pool size is two then you would get a 50 by 50.
That's down sampling by 2.
Let's go over mechanically how this works and then we'll explain why it works and why we would want
to include it in our neural network
so there are two kinds of pooling Max pulling an average pooling choosing which one to use is as you
probably guessed a hyper pragmatic choice.
Let's talk about Max pooling first since I think this one's a bit more common.
The best way to see how it works is by example.
So if we start with an image let's say it's a four by four and contains these numbers one two three
four five six seven eight sixteen fifteen fourteen thirteen twelve eleven ten nine so just the numbers
one the sixteen but organized non sequentially what Max pulling will do is take each two by two square
of which there are four and just return the maximum value in each of those squares so the max of 1 2
5 6 is 6 the max of 3 4 7 and 8 is 8 the max of sixteen fifteen twelve eleven is sixteen and the max
of fourteen thirteen Ten Nine is fourteen so the output image is six eight sixteen fourteen
as you might have guessed average pulling a similar but instead of taking the max we take the average
so for the same input image we have the following the average of one two five and six is three point
five the average of three four seven and eight is 5.5 the average of sixteen 15 2011 is thirteen point
five and the average of thirteen fourteen Ten Nine is eleven point five.
So the output image is three point five five point five thirteen point five eleven point five
now that you know how pooling works.
Let's talk about why we should use it the first advantage of pooling is practical.
If we down sample the image the image shrinks and therefore we have less data to process less multiplications
to do and of course that will speed things up but the second and more important advantage has to do
with image processing itself recall this idea of translational and variance This is the idea that I
don't care where in an image the feature occurred.
I just care that it occurred for example.
Both you and I can recognize this as in a we also recognize this as an A.
It doesn't matter where I put the A on the screen so translational and variance is important from a
biological point of view and how we as humans learn.
We don't have to learn how to recognize A's at each point in our field of vision.
Once we know what something looks like we can recognize it.
No matter if it's up down left or right.
So what is Max pulling doing.
Well consider a small two by two box.
Remember that convolution it tells us whether or not a feature has been found.
It's a pattern finder.
So imagine that in this two by two box I have these flags pattern found not found not found and not
found.
So it's a high number and a small number and a small number and a small number.
The pattern being found.
Returns a higher number.
So when I take the max it says the pattern has been found.
That's good because it tells me that the pattern has been found without carrying where it was found.
Average pooling is the same idea but I think Max pooling is a little more intuitive in this regard.
As a side note keep in mind that in this lecture I've assumed that we're using a pool size of two and
that at each pooling layer we select boxes which are two spaces apart now pulling layers have some flexibility
in this regard.
First it's not necessary for the pool size to be the same in both directions.
For example you could look at a two by three box or a three by two box although generally that's unconventional.
Second is that it's possible for the boxes to overlap the hyper parameter that controls this is called
the stride.
It tells us how many spaces to move the pulling box for each output previously we looked at the situation
where we had a pool size of two by two and a stride was also two.
This is a pretty common scenario.
If you add a string of one then or two by two box would overlap.
This is not really too common.
So while you have these options which I want to make you aware of they are usually not changed too often.
So what you see in this lecture is what we do most often we use the pool size of two in each direction
and we have a string of two so that we don't have any overlapping boxes.
So now that you know what pooling does and why it works let's discuss why the convolution part of the
CNN is organized the way it is.
Why should we have a sequential pattern consisting of a convoy layer followed by a pooling layer followed
by a convoy layer followed by another pooling layer and so forth.
If you recall I told you earlier that one cool thing researchers discovered is that CNN are able to
learn features hierarchically so the initial layers end up learning basic strokes.
The next layer ends up learning individual facial features such as nose eyes and lips.
The next layer ends up learning whole faces
the key points to keep in mind is that after each pulling the image shrinks but the filter sizes generally
stay around the same comment filter sizes are 3 by 3 5 by 5 and 7 by 7.
Generally speaking they are much smaller than the input image but what happens as the image shrinks
as it passes through each layer.
Well let's suppose we start with a 32 by 32 image and we have four convolution and pulling layers.
Let's assume we do convolution and same mode so it doesn't change the size of the image and let's assume
all our convolution filters are size 3 by 3.
Although this doesn't come into play just yet.
In this case after the first convolution and pulling the image shrinks down to 16 by 16.
After the second convolution and pull in the image shrinks down to a by 8.
After the third convolution and pulling the image shrinks down a four by four.
And finally after a fourth convolution and pulling the image shrinks down to two by two what happens
after this is still a mystery.
So let's just leave that for now and focus on the image shrinking
as you can see if you take a three by three filter and place it over a 32 by 32 image it occupies only
a very small portion of the image.
In other words is looking for very tiny patterns to match like for example simple edges and strokes.
Then the image becomes size 16 by 16.
Now all of a sudden in the three by three filter takes up four times as much space in the image it takes
up twice as much space in each direction and two times to his four.
In any case it occupies a much larger portion of the image relative to the first filter.
So now it's looking for patterns which take up more space on the image patterns like nose eyes and lips.
Next we get to an eight by eight image a three by three filter occupies almost a quarter of the image
after that convolution and pulling.
We get a four by four image now a three by three filter takes up most of the image.
So what have we learned.
We've learned that in a convolution on their own that work convolution and pooling results in two things
which are corollaries of each other first that the image input shrinks.
Second since filters generally stay the same size the filter looks for larger and larger patterns in
image.
This is what leads to the CNN learning hierarchical features of the input.
It first learns things that are relatively small compared to the total size of the image.
Then it learns to find bigger patterns and bigger patterns and so forth.
The last concepts regarding the convolution of layers we have to think about is are we not losing information
at every layer.
If we shrink the image and the answer is yes that's true.
We are losing spatial information because the image keeps shrinking.
We're saying we don't care where in the image the feature was found just that it was found somewhere
in the image.
So we're losing information about where it was.
But there's another component of the convolution process we haven't considered yet which is the number
of feature maps.
Again we have a general pattern to follow which is that while the size of the image generally decreases
the number of feature maps generally increases.
In other words I don't care where the feature is found.
I just care that it was found but I do care about different possible features.
I could find.
So while you're losing spatial information you're gaining information in terms of what features were
found in the image
one last note on this convolution and pulling pattern.
One of the hardest things to grasp for new students and deep learning is that we have so many choices.
We call these hyper parameters previously for a minute.
We looked at the learning rate the number of hidden layers and the number of hidden units.
But now with CNN is we have a ton of more choices with convolution.
We have to choose the filter size.
We have to choose the number of feature maps we have to choose the pool size and so forth.
But luckily I think this is one instance where there is a general pattern that most people follow.
So you're not going in completely blind so stick with these guidelines and you can be sure that you're
at least doing things per accepted convention that is choosing small filters relative to the image like
three by three five by five or seven by seven.
Another guideline is to repeat the pattern convolution followed by pooling.
Another guideline is to increase the number of feature maps at each convolution.
So you might start with 32 and then 64 then 128 and maybe 128 again.
The best way to learn about how other people are doing it is to read lots of papers and check out which
hyper parameters they selected.
Generally you'll find that this is the pattern followed by most convolution or networks
next.
I want to mention something a little funny but in fact pooling is sometimes not what we are going to
end up using in our CNN.
Researchers have found that we can do something similar but which is more efficient and sometimes works
just as well.
In particular recall that for pooling we have this concept of stride that tells us how far apart each
box should be when we take the max.
In fact convolution has a stride option as well so in this animation you can see how convolution works
with Stride.
Remember convolution just means multiply an ad with a sliding window when we have convolution with a
stripe parameter.
The stride tells us how far apart each window should be.
And of course our intuition tells us that if we use a strike of two then the output image length will
be half of what the input was.
In other words we get the same reduction in size by using a striking convolution instead of convolution
followed by pooling
the intuition behind why this works is that if you consider what an image looks like an image is just
large patches of stuff.
What I mean by that is each pixel is likely to have a similar value to its neighboring pixels.
For example suppose I'm looking at a red car.
I find a red pixel.
Now ask yourself is the pixel above that also red.
Yes.
Is the pixel below that also probably red.
Yes.
The same can be said of the pixel to the left and to the right.
That's just the nature of images pixels near each other are often very highly correlated.
You won't have this random jumping around of pixel values because something like that wouldn't look
like an image at all.
That would probably just be noise and so using a stride of two simply means having your filter skipping
every other pixel.
It's saying we don't care what those skipped pixels are because they probably are very close to the
pixels we were already looking at.
So to summarize the convolution part of our convolution on their own that work now looks like this.
We can have a series of static convolutions all with approximately the same filter size with increasing
feature maps at each layer.
Or we can have a series of convolutions followed by pooling again all with approximately the same filter
size and with increasing numbers of feature maps at each layer
the last part of this lecture will focus on the second half of the CNN which is the feed for neural
network part.
Now there's not much to discuss here except that we have to consider the shape of an image is not appropriate
for a dense field for no network.
Remember that when an image comes out of a convolution it's going to be three dimensional height by
width by number of feature maps but a field for a neural network takes in a feature vector a one dimensional
object.
Luckily we've already discussed how this works.
We can turn a 3D object into a one B object using the view function in PI torch
one alternative to the flan layer which I really like is the global Max pooling layer.
This is a little different from the regular kind of pooling we discussed earlier consider the question.
What if we have different sized images.
The largest source of images is the internet.
Generally speaking images we find on the Internet are not all the same size so it makes sense to wonder
is it possible to build a convolution or known network that is capable of handling different image sizes.
And the answer is yes.
Using a global Max pooling layer
Let's consider first why a simple flat in layer would not work.
Suppose again we have an input image of size 32 by 32 and for convolutions assume again we use convolution
and same mode.
And we have a stride of two.
So after the first convolution We get a 16 by 16.
After the second convolution We get an eight by eight.
After the third convolution We get a four by four.
And after the fourth convolution We get a two by two.
But now let's say we have an input image of size 64 by 64.
In this case the output after four convolutions would be four by four.
Let's say for simplicity's sake the number of output feature maps is one hundred then for the 32 by
32 image.
Doing a flat in operation would yield an output of size two by two by one hundred two times two times
100 which is four hundred four sixty four by sixty for image.
Doing a flat in operation would yield an output size four times four times 100 which is sixteen hundred.
And of course a feat for a neural network is not capable of handling vectors of different sizes.
Now let's consider how global mass pooling works.
The idea is simple if our input image is h by W. by C then the output is one by one by C or equivalently
just a vector of size C.
In other words it takes a max over the entire image along the spatial dimensions over each feature map.
Since we have C feature maps we end up with what is essentially a one dimensional vector of size C since
the 1 dimensions are redundant.
This follows the same general idea of pooling where we're saying I don't care where in the image the
feature was found just that it was found somewhere.
Global mass pooling is the extreme of this saying that the feature can be found anywhere in the entire
image across the entire height and width since the output is always a vector of size C it does not depend
on the size of the image meaning that it allows the network to handle images of any size.
As a side note we also have global average Pooley which is just the global version of regular average
pulling.
The only exception to this is if your images are too small in which case you'll just get an error when
you try to run your code.
For example if your image starts out as a two by two then it's not possible to reduce the dimensions
by half four times.
You'll notice this happens if you do something like add too many convolutions to your neural network.
All right so let's summarize the basic architecture of a modern convolution all knowing that we're the
first stage is a series of stride convolutions or optionally convolution followed by pooling and repeating
that pattern in the interface between convolution layers and dense layers.
We have a flatten.
We also optionally have a global Max pooling that reduces the image to just the max for each feature
map.
Finally once we flattened our data into a one dimensional vector we have a regular feed for a neural
network made up of a series of dense layers as usual and as you learned in the previous section the
activation and number of nodes for the final layer depends on the type of task being done.
So if you're doing scalar regression then you'll only have one output node with no activation function.
If you're doing binary classification you could have one output node with a sigmoid.
However for k equals two or more classes you can have K output nodes with a soft Max activation.