Now that you understand how exactly a convolution layer works, including the bias term activation function,
we can now consider the architecture of a convolutional neural network.
And why is that way?
So as a little bit of a history lesson, modern CNN's essentially all originated from the same model,
Alina.
This is named after Yann LeCun, one of the original deep learning pioneers, along with Geoffrey Hinton
and Joshua Bengio.
What we're going to do in this lecture is just start with what the architecture is and then go through
it piece by piece so that you understand why it is the way it is.
OK, so let's look at a typical CNN.
A typical CNN has two stages.
The first stage is a series of convolutional layers.
Importantly, these convolutional layers are usually followed by pooling layers, which is another type
of layer we'll discuss shortly.
The second stage is a series of dense layers, also called the fully connected layers.
These are just a regular feed for a neural network, as you saw in the previous section.
If you recall, I said in the previous section that neural networks have this hierarchical structure.
Each layer is always looking at features that were found in the previous layer.
So you can even think of the first stage as just another feature, Transformer.
It's a feature transformer that specifically works on images in finds image features.
Then once it's found those image features, it passes those into a neural network and then the neural
network does what it does best nonlinear classification or regression.
So let's have a closer look at the convolutional stage.
Now I just added a new layer without really explaining it the pooling layer.
Why would we want to do that at a high level, pooling acts as a downsampling operation?
What is downsampling?
That's when you make a smaller image out of a bigger image.
So imagine your image is 100 by 100.
After pooling, if the pool size is too, then you would get a 50 by 50.
That's downsampling by two.
Let's go over it mechanically how this works, and then we'll explain why it works and why we would
want to include it in our neural network.
So there are two kinds of pull in Max, pulling an average pull in choosing which one to use is, as
you probably guessed, a hyper parameter choice.
Let's talk about max pooling first, since I think this one's a bit more common.
The best way to see how it works is by example.
So if we start with an image, let's say it's a four by four and contains these numbers one two three
four five six seven eight 16 15 14 13 12 11 10 nine.
So just the numbers one to 16, but organized nine sequentially.
What Max Pooling will do is take each two by two square of which there are four and just return the
maximum value in each of those squares.
So the max of one two five six is six, the max of three, four, seven and eight is eight, the max
of 16, 15, 12, 11 is 16 and the max of 14, 13, 10, nine is 14.
So the output image is six, eight, 16, 14.
As you might have guessed, average pulling a similar, but instead of taking the max, we take the
average.
So for the same input image, we have the following the average of one two, five and six is three point
five.
The average of three, four seven and eight is five point five.
The average age of 16 15, 12 11 is thirteen point five.
And the average of 13 14 10 nine is eleven point five.
So the output image is three point five five point five thirteen point five eleven point five.
Now that you know how pooling works, let's talk about why we should use it.
The first advantage of pooling is practical.
If we downsample the image, the image shrinks and therefore we have less data to process, less multiplications
to do.
And of course, that will speed things up.
But the second and more important advantage has to do with image processing itself.
Recall this idea of translational invariance.
This is the idea that I don't care where in an image the feature occurred, I just care that it occurred.
For example, both you and I can recognize this as an error.
We also recognize this as an error.
It isn't matter where I put the air on the screen, so translational and variance is important from
a biological point of view and how we as humans learn.
We don't have to learn how to recognize A's at each point in our field of vision.
Once we know what something looks like, we can recognize it, no matter if it's up, down, left to
right.
So what is Max pooling doing?
Well, consider a small two by two box.
Remember that convolution tells us whether or not a feature has been found.
It's a pattern finder.
So imagine that in this two by two box, I have these flags pattern found, not found, not found,
not found.
So it's a high number and a small number and a small number and a small number.
The pattern being found returns a higher number.
So when I take the max, it says the pattern has been found.
That's good because it tells me that the pattern has been found without caring where it was found.
Average pulling is the same idea, but I think max pooling is a little more intuitive in this regard.
As I say, no, keep in mind that in this lecture, I assume that we're using a pool size of two and
they're at each pulling layer.
We select boxes, which are two spaces apart.
Now, pulling layers have some flexibility in this regard.
First, it's not necessary for the pool size to be the same in both directions.
For example, you can look at a two by three box or a three by two box, although generally that's unconventional.
Second is that it's possible for the boxes to overlap.
The type of parameter that controls this is called the stride.
It tells us how many spaces to move the pulling box for each output.
Previously, we looked at the situation where we had a pool size of two by two and the stride was also
two.
This is a pretty common scenario.
If you had to straight of one, then hour two by two box would overlap.
This is not really to comment.
So while you have these options, which I want to make you aware of, they're are usually not changed
too often.
So what you see in this lecture is what we do most often.
We use a pool size of two in each direction and we have a stride of two so that we don't have any overlapping
boxes.
So now that you know what pooling does and why it works.
Let's discuss why the convolutional part of the CNN is organized the way it is.
Why should we have a sequential pattern consisting of a conveyor followed by a pooling layer, followed
by a conveyor followed by another pulling layer and so forth?
If you recall, I told you earlier that one cool thing researchers discovered is that CNN's are able
to learn features hierarchically, so the initial layers end up learning basic strokes.
The next layer ends up learning individual facial features such as nose, eyes and lips.
The next layer ends up learning whole faces.
The key point to keep in mind is that after each pulling, the image shrinks, but the filter sizes
generally stay around the same comment.
Filter sizes are three by three, five by five and seven by seven.
Generally speaking, they are much smaller than the input image.
But what happens as the image shrinks as it passes through each layer?
Well, let's suppose we start with a 32 by 32 image and we have four convolutional and pulling layers.
Let's assume we do convolution and same mode, so it doesn't change the size of the image.
And let's assume all our convolution filters are size three by three.
Although this doesn't come into play just yet.
In this case, after the first convolution and pulling, the image shrinks down to 16 by 16.
After the second convolution and pull in, the image shrinks down to eight by eight.
After the third convolution and pulling them, it shrinks down to four by four.
And finally, after a fourth convolution are pulling.
The image shrinks down to two by two.
What happens after this is still a mystery.
So let's just leave that for now and focus on the image shrinking.
As you can see, if you take a three by three filter and place it over a 32 by 32 image, it occupies
only a very small portion of the image.
In other words, it's looking for very tiny patterns to match.
Like, for example, simple edges and strokes.
Then the image becomes size 16 by 16.
Now, all of a sudden, the three by three filter takes up four times as much space in the image.
It takes up twice as much space in each direction, and two times two is four.
In any case, it occupies a much larger portion of the image relative to the first filter.
So now it's looking for patterns which take up more space on the image patterns like nose, eyes and
lips.
Next, we get to an eight by eight image, a three by three filter occupies almost a quarter of the
image.
After that conversation and pulling, we get a four by four image now, a three by three filter takes
out most of the image.
So what have we learned?
We've learned that in a convolutional neural network, convolution and pulling results in two things
which are corollaries of each other.
First, the image input shrinks.
Second, since filters generally stay the same size, the filter looks for larger and larger patterns
in the image.
This is what leads to the CNN learning hierarchical features of the input.
It first learns things that are relatively small compared to the total size of the image.
Then it learns to find bigger patterns and bigger patterns and so forth.
The last concepts regarding the convolutional layers we have to think about is, are we not losing information
at every layer?
If we shrink the image and the answer is yes, that's true, we are losing spatial information because
the image keeps shrinking.
We're saying we don't care where in the image the feature was found, just that it was found somewhere
in the image.
So we're losing information about where it was.
But there's another component of the convolution process we haven't considered yet, which is the number
of feature maps.
Again, we have a general pattern of follow, which is that while the size of the image generally decreases,
the number of feature maps generally increases.
In other words, I don't care where the feature is found.
I just care that it was found.
But I do care about different possible features I could find.
So while you're losing spatial information, you're gaining information in terms of what features were
found in the image.
One last note on this convolution and pooling pattern.
One of the hardest things to grasp for new students in deep learning is that we have so many choices.
We call these hyper parameters previously for Ian is we looked at the learning rate, the number of
hidden layers and the number of hidden units.
But now with scenes, we have a ton of more choices.
With convolution, we have to choose the filter size.
We have to choose the number of feature maps.
We have to choose the pool size and so forth.
But luckily, I think this is one instance where there's a general pattern that most people follow,
so you're not going in completely blind.
So stick with these guidelines, and you can be sure that you're at least doing things per accepted
convention that is choosing small filters relative to the image like three by three, five by five or
seven by seven.
Another guideline is to repeat the pattern convolution, followed by pulling.
Another guideline is to increase the number of feature maps at each convolution.
So you might start with 32 and then 64 and 128, and maybe 128 again.
The best way to learn about how other people are doing it is to read lots of papers and check out which
hyper parameters they selected.
Generally, you'll find that this is the pattern followed by most convolutional networks.
Next, I want to mention something a little funny, but in fact, pooling is sometimes not what we are
going to end up using in our scenes.
Researchers have found that we can do something similar, but which is more efficient and sometimes
works just as well in particular.
Recall that for pooling, we have this concept of stride that tells us how far apart each box should
be when we take the max.
In fact, convolution has a stride option as well.
So in this animation, you can see how convolution works with stride.
Remember, convolution just means multiply an ad with a sliding window when we have convolution with
a stripe parameter.
The stride tells us how far apart each window should be.
And of course, our intuition tells us that if we use a stride of two, then the output image length
will be half of what the input was.
In other words, we get the same reduction in size by using a striated convolution instead of convolution,
followed by pulling.
The intuition behind why this works is that if you consider what an image looks like, an image is just
large patches of stuff.
What I mean by that is each pixel is likely to have a similar value to its neighboring pixels.
For example, suppose I'm looking at a red car, I find a red pixel.
Now ask yourself, is the pixel above that also red?
Yes.
Is the pixel below that also probably red?
Yes.
The same can be said of the Pixel to the left and to the right.
That's just the nature of images.
Pixels near each other are often very highly correlated.
You won't have this random jumping around of pixel values because something like that wouldn't look
like an image at all.
That would probably just be noise.
And so using a stride of two simply means having your filter skipping every other pixel.
It's saying we don't care what those two pixels are because they probably are very close to the pixels
we were already looking at.
So to summarize, the convolutional part of our convolutional neural network now looks like this, we
can have a series of shredded conversations, all with approximately the same filter size with increasing
feature maps at each layer.
Or we can have a series of convolutions, followed by pooling again, all with approximately the same
filter size and with increasing numbers of feature maps at each layer.
The last part of this lecture will focus on the second half of the CNN, which is the feedforward neural
network part.
Now, there's not much to discuss here, except that we have to consider the shape of an image is not
appropriate for a dance feed, for no network.
Remember that when an image comes out of a competition, it's going to be three dimensional height by
width by number of feature maps.
But a field for a neural network takes in a feature vector a one dimensional object.
Luckily, we've already discussed how this works.
We can turn a three dimensional object into a one dimensional object using the flattened layer and caress.
One alternative to the flag earlier, which I really like, is the global max pooling layer.
This is a little different from the regular kind of pooling we discussed earlier.
Consider the question What if we have different sized images?
The largest source of images is the internet.
Generally speaking, images we find on the internet are not all the same size.
So it makes sense to wonder is it possible to build a convolutional neural network that is capable of
handling different image sizes?
And the answer is yes.
Using a global max pooling layer.
Let's consider first why a simple, flat layer would not work.
Suppose again, we have an input image of size 32 by 32 and full convolutions.
Assume again, we use convolution and same mode, and we have a stride of two.
So after the first convolution, we get a 16 by 16.
After the second convolution, we get an eight by eight.
After the third convolution, we get a four by four and after the fourth convolution, we get a two
by two.
But now let's say we have an input image of CI 64 by 64.
In this case, the output after four convolutions would be four by four, let's say for simplicity's
sake, the number of output feature maps is 100.
Then for the 32 by 32 image, doing a flattening operation would yield an output of size two by two
by one hundred two times two times 100, which is 400 for 64 by 64.
Image doing a flattened operation would yield an output size of four times four times 100, which is
6500.
And of course, a feedforward neural network is not capable of handling vectors of different sizes.
Now, let's consider how global max pooling works.
The idea is simple if our input image is H by W by C, then the output is one by one by C or equivalently
just a vector of size C.
In other words, it takes the MAX over the entire image along the spatial dimensions over each feature
map.
Since we have C feature maps, we end up with what is essentially a one dimensional vector of size C
since the one dimensions are redundant.
In fact, TensorFlow and Keras get rid of the redundant dimensions for us.
This follows the same general idea of pooling where we're saying I don't care where in the image the
feature was found just that it was found somewhere.
Global max pooling is the extreme of this, saying that the feature can be found anywhere in the entire
image across the entire height and width, since the output is always a vector of size.
It does not depend on the size of the image, meaning that it allows the network to handle images of
any size.
As a side note, we also have global average pulling, which is just the global version of regular average
pulling.
The only exception to this is if your images are too small, in which case you'll just get an error
when you try to run your code.
For example, if your image starts out as a two by two, then it's not possible to reduce the dimensions
by half four times.
You'll notice this happens if you do something like add too many convolutions to your neural network.
All right, so let's summarize the basic architecture of a modern convolutional neural network.
The first stage is a series of Strider convolutions or optionally convolution, followed by pooling
and repeating that pattern in the interface between convolution layers and dense layers.
We have a flattened.
We also optionally have a global max pooling that reduces the image to just the MAX for each feature
map.
Finally, once we flattened our data into a one dimensional vector, we have a regular feed for a neural
network made up of a series of dense layers as usual.
And as you learn in the previous section, the activation and number of nodes for the final layer depends
on the type of task being done.
So if you're doing scalar regression, then you'll only have one output node with no activation function.
If you're doing binary classification, you could have one output node with a sigmoid.
However, for K equals two or more classes, you can have kapwa nodes with a soft max activation.