Now that you understand how exactly a convolution layer works, including the bias term and activation
function, we can now consider the architecture of a convolutional neural network and why it's that
way.
So as a little bit of a history lesson in modern science, essentially all originated from the same
model Ilina.
This is named after Yann Lican, one of the original deep learning pioneers, along with Geoffrey Hinton
and Joshua Bengoa.
What we're going to do in this lecture is just start with what the architecture is and then go through
it piece by piece so that you understand why it is the way it is.
OK, so let's look at a typical CNN, a typical CNN has two stages.
The first stage is a series of convolutional layers.
Importantly, these convolutional layers are usually followed by pooling layers, which is another type
of layer we'll discuss shortly.
The second stage is a series of dense layers, also called a fully connected layers.
These are just a regular feed for neural network, as you saw in the previous section.
If you recall, I said in the previous section that neural networks have this hierarchical structure.
Each layer is always looking at features that were found in the previous layer.
So you can even think of the first stage as just another feature transformer.
It's a feature transformer that specifically works on images and finds image features.
Then once it's found those image features, it passes those into a neural network and then the neural
network does what it does best, nonlinear classification or regression.
So let's have a closer look at the convolutional stage.
Now, I just added a new layer without really explaining it, the pooling layer.
Why would we want to do that at a high level, pooling access, a downsampling operation?
What is downsampling.
That's when you make a smaller image out of a bigger image.
So imagine your images 100 by 100 after pulling.
If the pool size is two, then you would get a 50 by 50.
That's down sampling by two.
Let's go over mechanically how this works and then we'll explain why it works and why we would want
to include it in our neural network.
So there are two kinds of pooling, Max, pulling an average pooling, choosing which one to use is,
as you probably guessed, a hyper pragmatic choice.
Let's talk about Max pooling first, since I think this one's a bit more common, the best way to see
how it works is by example.
So if we start with an image, let's say it's a four by four and contains these numbers, one, two,
three, four, five, six, seven, eight, 16, 15, 14, 13, 12, 11, 10, nine.
So just the number one to 16, but organized nonsequential.
What Max will do is take each two by two square of which they're are for and just return the maximum
value in each of those squares.
So the max of one, two, five, six is six, the max of three, four, seven and eight is eight, the
max of 16, 15, 12, 11 is 16 and the max of 14, 13, 10, nine is 14.
So the output image is six, eight, 16, 14.
As you might have guessed, average pulling a similar, but instead of taking the max, we take the
average.
So for the same input image, we have the following the average of one to five and six is three point
five.
The average of three, four, seven and eight is five point five.
The average of sixteen, fifteen, twelve, eleven is thirteen point five and the average of thirteen,
fourteen, ten, nine is eleven point five.
So the output image is three point five five point five thirteen point five eleven point five.
Now that you know how polling works, let's talk about why we should use it.
The first advantage of polling is practical.
If we downsampled the image, the image shrinks and therefore we have less data to process, less multiplications
to do.
And of course, that will speed things up.
But the second and more important advantage has to do with image processing itself.
Recall this idea of translational invariance.
This is the idea that I don't care where in an image the feature occurred, I just care that it occurred.
For example, both you and I can recognize this as an error.
We also recognize this as an error.
It doesn't matter where I put the ad on the screen.
So translational invariance is important from a biological point of view and how we as humans learn.
We don't have to learn how to recognize Azz at each point in our field of vision.
Once we know what something looks like, we can recognize it no matter if it's up, down, left or right.
So what is Max Pooling doing?
Well, consider a small to buy two box, remember that convolution it tells us whether or not a feature
has been found.
It's a pattern finder.
So imagine that in this to buy two box.
I have these flags Pat and found.
Not found.
So it's a high number and a small number.
And a small number.
The pattern being found returns a higher number.
So when I take the max, it says the pattern has been found.
That's good, because it tells me that the pattern has been found without caring where it was found.
Average pulling is the same idea.
But I think Max pooling is a little more intuitive in this regard.
As a side note, keep in mind that in this lecture, I assumed that we're using a pool size of two and
that at each pulling layer we select boxes which are two spaces apart.
Now, pulling layers have some flexibility in this regard.
First, it's not necessary for the pool size to be the same in both directions.
For example, you could look at a two by three box or a three by two box, although generally that's
unconventional.
Second is that it's possible for the boxes to overlap.
The hyper parameter that controls this is called the stride.
It tells us how many spaces to move the pulling box for each output.
Previously, we looked at the situation where we had a pool size of two by two and the stride was also
too.
This is a pretty common scenario.
If you had to try of one, then our two by two blocks would overlap.
This is not really too common.
So while you have these options, which I want to make you aware of, they are usually not changed too
often.
So what you see in this lecture is what we do most often.
We use a pool size of two in each direction and we have a strike of two so that we don't have any overlapping
boxes.
So now that you know what pooling does and why it works, let's discuss why the convolutional part of
the CNN is organized the way it is.
Why should we have a sequential pattern consisting of a conveyor followed by a pooling layer, followed
by a conveyor, followed by another pooling layer and so forth?
If you recall, I told you earlier that one cool thing researchers discovered is that CNN's are able
to learn features hierarchically so the initial layers end up learning basic strokes.
The next layer ends up learning individual facial features such as nose, eyes and lips.
The next layer ends up learning whole faces.
The key point to keep in mind is that after each pulling, the image shrinks, but the filter sizes
generally stay around the same come in.
Filter sizes are three by three, five by five and seven by seven.
Generally speaking, they are much smaller than the input image.
But what happens as the image shrinks, as it passes through each layer?
Well, let's suppose we start with a 32 by 32 image and we have four convolutional and pulling layers.
Let's assume we do convolution in and same mode so it doesn't change the size of the image.
And let's assume all our convolution filters are size three by three, although this doesn't come into
play just yet.
In this case, after the first convolution and pulling, the image shrinks down to 16 by 16 after the
second convolution and pull in, the image shrinks down to eight by eight.
After the third convolutional pulling the image shrinks down to four by four.
And finally, after a fourth convolution and pulling, the image shrinks down to two by two.
What happens after this is still a mystery?
So let's just leave that for now and focus on the image shrinking.
As you can see, if you take a three by three filter and place it over a 32 by 32 image, it occupies
only a very small portion of the image.
In other words, it's looking for very tiny patterns to match, like, for example, simple edges and
strokes.
Then the image becomes size 16 by 16.
Now, all of a sudden, the three by three filter takes up four times as much space in the image.
It takes up twice as much space in each direction and two times to his four.
In any case, it occupies a much larger portion of the image relative to the first filter.
So now it's looking for patterns which take up more space on the image patterns like nose, eyes and
lips.
Next, we get to an eight by eight image, a three by three filter occupies almost a quarter of the
image.
After that conversation and pulling, we get a four by four image now, a three by three filter takes
up most of the image.
So what have we learned?
We've learned that in a convolutional neural network, convolution and pulling results in two things
which are corollaries of each other.
First, the image input shrinks.
Second, since filters generally stay the same size, the filter looks for larger and larger patterns
in the image.
This is what leads to the CNN learning hierarchical features of the input.
It first learns things that are relatively small compared to the total size of the image.
Then it learns to find bigger patterns and bigger patterns and so forth.
The last concepts regarding the convolutional is we have to think about is are we not losing information
at every layer if we shrink the image?
And the answer is yes, that's true.
We are losing spatial information because the image keeps shrinking.
We're saying we don't care where in the image the feature was found, just that it was found somewhere
in the image.
So we're losing information about where it was.
But there's another component of the convolution process we haven't considered yet, which is the number
of feature maps.
Again, we have a general pattern to follow, which is that while the size of the image generally decreases,
the number of feature maps generally increases.
In other words, I don't care where the feature is found, I just care that it was found.
But I do care about different possible features I could find.
So while you're losing spatial information, you're gaining information in terms of what features were
found in the image.
One last note on this convolution and pulling pattern, one of the hardest things to grasp for new students
in deep learning is that we have so many choices.
We call these hyper parameters.
Previously, for instance, we looked at the learning rate, the number of hidden layers and the number
of hidden units.
But now with CNN, we have a ton of more choices with convolution.
We have to choose the filter size.
We have to choose the number of feature maps, we have to choose the pool size and so forth.
But luckily, I think this is one instance where there's a general pattern that most people follow.
So you're not going in completely blind.
So stick with these guidelines and you can be sure that you're at least doing things per accepted convention
that is choosing small filters relative to the image like three by three, five by five or seven by
seven.
Another guideline is to repeat the pattern convolutional, followed by Pouilly.
Another guideline is to increase the number of feature maps at each convolution.
So you might start with 32 and then 64 and 128 and maybe 128.
Again, the best way to learn about how other people are doing it is to read lots of papers and check
out which hyper parameters they selected.
Generally, you'll find that this is the pattern followed by most convolutional networks.
Next, I want to mention something a little funny, but in fact, poing is sometimes not what we are
going to end up using in our CNN.
Researchers have found that we can do something similar, but which is more efficient and sometimes
works just as well.
In particular, recall that for pooling, we have this concept of stride that tells us how far apart
each box should be when we take the max.
In fact, Convolution has a stride option as well.
So in this animation, you can see how convolution works with stride, remember, convolution just means
multiplying ad with a sliding window.
When we have convolution with a straight parameter, the stria tells us how far apart each window should
be.
And of course, our intuition tells us that if we use a string of two, then the output image length
will be half of what the input was.
In other words, we get the same reduction in size by using a striated convolution instead of convolution,
followed by Pouilly.
The intuition behind why this works is that if you consider what an image looks like, an image is just
large patches of stuff.
What I mean by that is each pixel is likely to have a similar value to its neighboring pixels.
For example, suppose I'm looking at a red car, I find a red pixel.
Now, ask yourself, is the pixel above that also red?
Yes.
Is the pixel below that also probably red?
Yes.
The same can be said of the pixel to the left, ends of the right.
That's just the nature of images.
Pixels near each other are often very highly correlated.
You won't have this random jumping around of pixel values because something like that wouldn't look
like an image at all.
That would probably just be noise.
And so using a string of two simply means having your filter, skipping every other pixel.
It's saying we don't care what those pixels are because they probably are very close to the pixels we
were already looking at.
So to summarize, the convolutional part of our convolutional neural network now looks like this, we
can have a series of strato convolutions, all with approximately the same filter size, with increasing
feature maps at each layer.
Or we can have a series of convolutions followed by pooling again, all with approximately the same
filter size and with increasing numbers of feature maps at each layer.
The last part of this lecture will focus on the second half of the CNN, which is the feed for neural
network part.
Now, there's not much to discuss here except that we have to consider the shape of an image is not
appropriate for a dense field for known that work.
Remember that when an image comes out of a convolution, it's going to be three dimensional height by
width, by a number of feature maps, but a feel for where it takes in a feature vector, a one dimensional
object.
Luckily, we've already discussed how this works.
We can turn a three dimensional object into a one dimensional object using the flattened layer in Kharas.
One alternative to the flannelette, which I really like, is the global max pooling there, this is
a little different from the regular kind of pooling we discussed earlier.
Consider the question, what if we have different sized images?
The largest source of images is the Internet.
Generally speaking, images we find on the Internet are not all the same size.
So it makes sense to wonder is it possible to build a convolutional network that is capable of handling
different image sizes?
And the answer is yes.
Using a global max, pooling their.
Let's consider first why a simple, flattened layer would not work.
Suppose again, we have an input image of size 32 by 32 and four convolutions.
Assume again, we use convolution and say mode and we have a straight up to so after the first convolution,
we get a 16 by 16.
After the second convolution, we get an eight by eight.
After the third convolution, we get a four by four.
And after the fourth convolution, we get a two by two.
But now let's say we have an input image of, say, 64 by 64.
In this case, the output after four convolutions would be four by four.
Let's say for simplicity's sake, the number of output feature maps is one hundred.
Then for the 32 by 32 image, doing a flattened operation would yield an output of size, two by two
by one hundred two times, two times 100, which is four hundred for 64 by 64 image doing a flattened
operation would yield an output size four times, four times 100, which is six hundred.
And of course, a feat for a neural network is not capable of handling vectors of different sizes.
Now, let's consider how global mass pooling works.
The idea is simple if our input image is by WBC, then the output is one by one by sea or equivalently
just a vector of Sisay.
In other words, it takes the max over the entire image, along the spatial dimensions, over each feature
map.
Since we have feature maps, we end up with what is essentially a one dimensional vector of Sisay since
the one dimensions are redundant.
In fact, tents flown keris get rid of the redundant dimensions for us, this follows the same general
idea of pooling where we're saying I don't care where in the image the feature was found, just that
it was found somewhere.
Global mass pooling is the extreme of this, saying that the feature can be found anywhere in the entire
image across the entire height and width.
Since the output is always a vector of Sisay, it does not depend on the size of the image, meaning
that it allows the network to handle images of any size.
As a side note, we also have global average pully, which is just the global version of regular average
Pouilly.
The only exception to this is if your images are too small, in which case you'll just get an error
when you try to run your code.
For example, if your image starts out as a two by two, then it's not possible to reduce the dimensions
by half four times.
You'll notice this happens if you do something like add too many convolutions to your neural network.
Now, for most of this lecture we've been discussing, CNN's in the context of images, which is more
natural and it's also where CNN's originated.
We could use these four time series, but a more direct way is to use one's convolutions.
So just for completion sake, let's look at the shape progression for a time series using one D convolutions.
So suppose that we start with some multivariate or univariate time series of shape TBD.
If it's UNIVARIATE, then D is just equal to one.
Let's also suppose for simplicity that C is equal to one twenty eight.
The next step is to pass this through some convolution layer.
Suppose that we use a filter of like three with thirty two output feature maps.
Let's also suppose for simplicity that we are using same mode convolution.
In this case, the output of this convolutional layer will be of size T by thirty two or one, twenty
eight by thirty two.
The next step is to pass this through a pooling layer with the pool size of two and a straight of two.
This will cut the time dimension in half, so the result is now sixty four by thirty two.
The next step is to pass this through another convolution.
Suppose that this again is same mode convolution with a filter size of sixty for the output of this
layer will be sixty four by sixty four.
We again pass this through a pooling layer, which reduces the time dimension by half.
Once again, the output of this is thirty two by sixty four.
Let's suppose that we have one final convolution there with one hundred twenty eight feature maps and
everything else, the same as before, the output of this layer will be thirty two by one, twenty eight.
Now, let's suppose at this point we're ready to apply a dense neural network.
So instead of regular pooling, we're going to apply a global max pooling.
As you recall, global max pooling reduces the time dimension to one.
So the output of this will be one by one twenty eight or equivalently, just one twenty eight.
After this, we passed the 128 dimensional feature vector as normal through a regular end.
OK, so hopefully that was easy.
It's the same as a CNN four images just with one less dimension.
All right, so let's summarize the basic architecture of a modern, convolutional neural network.
The first stage is a series of striated convolutions or optionally convolution, followed by pooling
and repeating that pattern in the interface between convolution layers and dense layers.
We have a flat and we also optionally have a global max pooling that reduces the image to just the max
for each feature map.
Finally, once we flattened our data into a one dimensional vector, we have a regular feed for a neural
network made up of a series of dense layers, as usual.
And as you learn in the previous section, the activation and number of nodes for the final layer depends
on the type of task being done.
So if you're doing scalar regression, then you'll only have one output node with no activation function
if you're doing binary classification.
You could have one output node with a sigmoid.
However, four K equals two or more classes.
You can have Ka'aba nodes with a softmax activation.