Hello, everyone, and welcome to this new and exciting session in which we're going to visualize the
convolutional neural networks feature maps.
One very important part of building robust deep learning models involves understanding how these models
work or understanding what goes on in the different hidden layers.
And so in the section, we'll focus on taking a model which has already been pre trained and then generating
this feature maps.
So we get to see exactly what goes on under the hood.
The pre trained model will be using here will be the G 16.
So we'll simply copy this, get back to our code, paste this out.
There we go.
We have our G 16.
We are going to take the top, so we'll take all this off now our input shape and we'll take this input
tensor off.
We're not going to include the top, so we set this to false.
That's it.
Then this input shape will define it as our size here.
So we have configuration in size, that's fine.
And we add this year.
Okay, so that's it.
We set this and we give it this name, VG Backbone.
So we have VG backbone right here.
We can check out the summary VG Backbone summary.
Run that.
There we go.
You see, it has about 14.7 million parameters.
We now move on to the next step where we're going to create this other model which will permit us visualize
this feature maps.
Now to explain how this works.
Let's recall that we have this VG right here.
So we have our VG.
And then what the VG does is it takes in an input image.
So we have an input image and then it produces a single output.
Now if we have, say, not included the top, then we would have this output which is hit by it by 512.
So here we have eight by eight by 512.
And when it took in this 256 by 256 by three input.
Now since we have only this one single output.
And we are interested in visualizing the hidden layers that what that's what goes in the VG model.
What we'll do now is we'll create a new model.
We'll create a new model right here, which instead now has many different outputs.
And there's different outputs will come from this different hidden layers.
So we could take this one and it becomes now an output.
This one, it becomes an output, this one output, and so on and so forth.
So basically this hidden layers now are this hidden or the outputs of the hidden layers, that's our
feature maps will become our outputs.
So now have this model with this as input and then this as outputs.
So this will be now our different outputs instead of just a single output.
Now we have this different outputs.
There'll be about 17 outputs in total.
Now you may also decide to pick the specific output, so you may want to take only the conf layers or
only the outputs of the conf layer.
So you meet the max layers your your, your and your with this one.
But it all depends on you and we'll see how to do this.
So let's get back to the code where we're not going to build our feature maps.
So we'll take the feature maps and then we'll put this in a list so you will get the layer output and
this four layer in the VG backbone layers.
So we, we, we have this big backbone layers right here or this big backbone model here.
We're going to get all its layers starting from this one.
We're going to pick the input.
So we'll simply have this and that will be it.
So we take this layers and then from here we'll build this new model which we'll call feature map model
and Keras model from your it takes in as inputs the VG backbone.
So we have VG backbone input here.
We have the VG backbone.
Input is the same input, but now with the difference that the outputs is this feature maps here.
So we don't have this just one output, but all this other hidden layer outputs will now become our
outputs or be part of our outputs.
So here we have feature maps.
So that's it.
We've built this new model.
We'll view the summary feature map model, that summary, and now we go, So that's it.
It looks similar to what we have, but if we had picked, say, from model one to just model for or
sorry, from layer one to layer four, then you see it's shut in because this all we need for this,
our new model.
But since we're getting right up to the end, you see that we actually go through the whole VG model,
but with the difference that now we have outputs, that we have many outputs and we have just this one
single input unlike before, where we have one input and one output.
So from here we have this model which we've just this new model which we've just designed starts from
one run that again, you have that.
And now let's head on to passing an input to this model.
So what we want to do now is we take this input image and then we pass it into our model.
And now since our model outputs the different feature maps, the different hidden layer outputs, we
will now be able to visualize what's going on inside our VG model.
Now to get this output, we are going to use something similar to the testing which we've seen already.
Recall we did.
We carried out this testing here where we take we read this image.
We could simply copy this, where we read this image, and then we pass it to our model to get the output.
But now in our case, the model let's reduce this.
The model will be working with is our newly created feature map model.
So let's have this here and that's it.
We have our test image, we resize, we pass this in this feature map model, and then you'll see that
when we run this, we now check our feature maps.
So we'll say for let's say for I in range length of the feature maps.
We want to print out one to print out the F map ship.
So we have that list has no attribute shape.
Okay F maps I so let's pick out this I, we run that and you see this.
So you see that the output styles from your from this one instead of the input starts from actually
starts from here as we since we have decided to start from this one since we start from here because
we don't want to include the input as part of our output.
So that's logical.
We have that.
We start from this right up to this very last one year.
Now here we've picked the conf layers and the max layers.
So we have all this now so we can now visualize this different feature maps right here.
Now let's note this length, let's print this out.
Let's note this land.
You see you have 18 different outputs.
We get back here and we modify this.
So we'll say that we'll only do this if it's conf of layer.
That name is true.
So if this is true, if this is true, then we are going to attach this to the outputs.
Now we could also do this.
We could just simply have it like this and that's fine.
Now.
So what this means is we are only going to take the conf layers as part of our output.
Now we could define this is conf, this is context and a layer name.
So basically this layer names what we have here does you see why it's important to always give your
layers some names because now you see it's helpful or it's used now to differentiate between the different
types of layers.
So here we have this layer name and then we're going to see that if this layer name, if this layer
name or rather if conf in the layer name, we return true, then else we return false.
So we run that.
There we go, We run this again.
It looks similar to what we had before, but one thing you will notice now is that this length is reduced.
So we've taken off five.
We have gone from 18 to 13, so we've taken off five layers which correspond to this max layers since
this conf, this conf isn't in this name right here.
So that's it.
We have that.
We could also decide to say, okay, we want to take only the max polls.
So we will say if Paul or Paul we run that and check this line.
You see now only five.
There we go.
So that's it.
Let's get back to conf and we have that.
Okay, so now we've run this and we have the different ships now to carry out the final visualization.
You see you have this F maps here.
So we're going to go through each and every feature map.
So for I in range, the length of our feature maps, we now create this figure and then we specify the
fixed size.
So we have fixed size equal to 56 by 256.
Now we have that we call this method right here.
And then from year since we're going through each and every feature map is important for us to get the
feature size.
So we want to get this values for each feature map.
Now with this we just simply have F maps, we have K, and then I'll rather I here.
So we get in this, we pick a feature map, we get that and then we get the shape one.
So this will permit us to get this value.
So this is shape zero shape one shape to shape.
Three.
So we get this feature size though of feature map size.
Then we now get to this number of channels.
So we have N channels equal the feature maps.
I shape three three because the 0123.
So is how we get the number of channels.
Now we have this already set.
We want to be able to visualize this such that like your search that all these channels are aligned
on a single line.
So because we have this 512 16 by 16, let's let's check this earlier.
Once we have like your we have 64 to 56 by 256 images.
So let's suppose that this is one of them.
Yeah.
We have this 256 to 56 by 256 right here.
And then we have 64 of this for the 64 different channels right here.
Now what we want to do is take this one.
Let's take this one, put it here, take this other one and align it.
Take the next one, align it so we could visualize this in one line up to the very last one right here.
So to do this now, what we'll do is we'll create another array, which we'll call joint maps, Joint
maps or MP.
Once we initialize that way, and then the size is will take the feature size.
So we have f size and then to get the this for the width, this for the height actually.
So we know that the height in the case of 256 by 256 this height is 256.
So we have this height to 56, this distance to 56, but then now the width is going to change.
So the width is no longer 256 but 256 times in this case 64.
So here we have 256, here we have f size times the number of channels.
So that's how we do that.
So with that, we now have this joint maps which is initialized to one.
So we have this array now set.
Now the next step will be to fill in this values, this outputs, this features in this array.
Now we understand how this joint maps here was created.
We now go ahead to fill this information, all these different features in this giant maps array.
So, yeah, we'll do we'll go through the different channels.
So for J in range and channels.
So basically in channels we have that and then we fill in the joint maps.
Now the way we'll do this is we'll keep the height fixed so we have the height fixed and then in this
width dimension will fill this information in a way that as we go from one channel to another, we are
going to skip steps up to 56.
So here we will have f size, our filter size, our feature map sizes, 256 F size times g, and then
we go to F size times G plus one.
So what this means is would we fix the height?
As we've said already, we will fix this height here.
The height is fixed to 56.
That's it here.
So we'll take all elements and the height and then for the width.
When J equals zero, for example, we have zero up to zero plus one.
So the year we have zero zero plus one, so we have zero.
And then zero times F is zero, then one times F size is 256.
So this means that in the width dimension we're going to go in the height we have to 56 and the width
we have to 56.
Now when J goes to one now here we have one one times f size is 256, so we have 256 and then one plus
one is two, two times 256 is 512.
So now we'll skip to 56 steps at this point.
We add this point now and we get this one and then we repeat this same process again.
When when J equals two, we have 512 and then we move to this should be 768 no two, five, six here,
768 So now we go from 512 to 768.
And so that's how we are going to be filling up this different positions right here.
Now, once we have this already set, we now go ahead and pass in the data what we have to fill in here.
So we have F maps and then we take in I if we consider this case where G equals zero, that's we've
picked this zone from zero 2 to 56 and we've collected all the height.
Then we have this patch right here and to fill up this patch we have our feature maps which we've seen
already.
But then we are going to take a particular feature map.
Obviously this either picks that particular feature map, then once we pick the particular feature map,
we can now go ahead and set this here to the values of the feature maps while selecting the particular
channel.
So now we when G equals zero, for example, we take the zero channel and so we take all the values
which come before and then pick out now this zero channel, for example.
Now here is G.
So that's it.
We have, we have our join maps which has now been created and then we could take this off how we now
are ready to plot our image.
So we have this show and then we passenger and maps.
Now if we want to pass all this is going to be very rum consuming.
So we just take the we select all the height and then we'll pick some value.
So we'll go from zero to, for example, 512.
So we'll have that.
And now we could run this, but before running this we need to set the different axis.
Here we have axis, then we have plot, dot subplot, the length of the feature maps.
So yeah, we will basically if we have 13 feature maps, then we are going to have this different subplots.
Your one.
And then here we have I plus one.
So that is it.
So once we have this set, we can now run this and see what we get.
Takes a while to run and now that is complete.
We could visualize the results here.
Let's simply scroll down and you see this result.
You see for the initial layers, we have this low level features which have been extracted.
So we have we could see this clearly here.
And as we go down or as we go further or deeper into the network, this the features we extracting start
to become more high level features.
So you see this see here, see this one focuses see here we extract this mount here unlike before,
where we're more focused on edges.
So that's it.
We keep going deeper and we see the outputs or the results we get in and that's it.
We have just visualized a train models feature maps.
Now another thing we could do is suppose at this beginning here that we have known right here.
So we don't we don't want the pre-trained weights, so we run this, run this again and check out on
what we are going to or what the model is going to produce.
Here is where we get you.
Let's crawl.
So you get to see this.
You see the inputs.
And as we go deeper.
We will notice that not much information is yet to be extracted from the inputs.