Hello, everyone, and welcome to this new and exciting session in which we are going to see how to
implement the great cam technique which permits us visually explain how the deep neural networks work.
This was first developed in this paper by Ramprasad et al and entitled Grant come a visual explanations
from deep networks via gradient based localization.
So as you could see here, we have this input.
For example, let's reduce this, we have this original image and then we can see this grad cam output.
Now the task here is a classification task where we're trying to say whether there's a cat or dog in
the image and then with a grad cam for the cat were able to detect the portion of the image which influenced
the model to say that there is a cat in the image.
And also for the grad cam dog, we have this part here which shows us that this portion influences the
model the most in detecting or seeing that there is a dog in the image.
Now we also have the grad cam for different models.
You see this grad cam for the VG model, and we have this Reznick grad Cam right here.
So we see that two different models can produce two different grad cams, though generally they should
be similar.
We see that with the rest net we have this larger surface as compared to this VG And we could also see
this for the cat.
Now that said, we are going to implement the grab cam technique which generates our, which permits
are to generate these kinds of visualizations which tell us what parts of the input influence the model
in taking certain decisions.
As we can see in this figure two from the paper, given an image and a class of interest, for example,
the tiger cat or any other type of differentiable output.
So like here we have this image, you see the image and we're giving the class tiger cat.
So yeah, the task is classic image classification.
So here we have this class tiger cat.
Now, for now, we'll keep our this here, we'll focus only on this image classification.
So essentially we have this image, as we've said already, we we have the class, then we pass this
through a CNN.
That's a convolutional neural network.
You have that right here where let's say we forward propagate the image to the scene part of the model.
Remember this, CNN has already been trained.
So with all this to the CNN part of the model and then through the text specific competitions to obtain
an raw score for the category.
So task specific here, in our case, our task is image classification.
So this year is specific to our task, which is that of image classification.
Now the gradients are set to zero for all classes except the desired class.
Then the signal is then back propagated to the rectified convolutional feature maps of interest.
So you can see from here that when we follow propagate this to our CNN, we have this output feature
maps.
Here you have rectified conf feature maps.
So these are feature maps and then we pass this through this fully connected layers.
Then we obtain the outputs from here, we bind, propagate, as we said already to this rectified feature
maps.
I could see that the result end of this back propagation is what we have here.
Notice how this is colored compared to this.
So the years of future maps here is the output from back propagation from this output backed up to this
feature maps.
So we we obtained this by doing the derivative of this output.
Let's say y, let's say y tiger, let's call it tc y tc y tiger cat.
With respect to x feature maps, let's call it FM.
So what we find in our we compute in this derivatives of the output with respect to each and every value
we have here for the feature maps.
And obviously this would give us a corresponding feature map, just like having let's, let's pick any
position, let's say this one on position here.
We take the output respect to this, we take the output respect to another position, to the output,
respect on that position and so on and so forth.
And by doing this, we reconstruct somehow this feature maps.
But this time around it's going to be the derivatives of the output respect to each and every position.
Then from here, as you could see, let's take this off from here.
What we do is we are going to obtain the mean for each and every one of those features.
So you see this one, this one does pinkish you have this here.
So we obtained the mean of for this we take for this we obtain this mean this and so on and so forth.
So we obtain all this and so all this mean values now are going to be multiplied by this feature maps.
So for this, for example, this one year we will take this and multiply this features.
We take this multiply.
Those features and so on and so forth.
Then once we get this, we are going to combine all this or add all these features up and then pass
this through a low function, which then produces our grad camp.
And so that's how this works in theory, that's how we obtain this grant.
Can we have here let's dive into the code and get this practically.
So first, since first we have this efficient net B five model, which we had trained already.
So we'll just load the weights.
Let's go ahead and load this weights.
So we have pre-trained model load weights and there we go from here, we could test out the model.
So we have this image.
Here is basically what we've seen already in testing.
So we have the test image, we expand dimensions and then we have the Pre-trained model, which now
takes in our image array.
So this is the output right here.
You could see that the max is at this index one, which is logical because it's a happy image.
We could see that right here.
The person is happy.
So this is our image and here's what our model produces.
Remember, as we've seen already, we have actually three classes.
We have angry, happy and sad.
And this is a zero index index one and the index two.
So our output is index one, meaning this prediction is correct.
At this point.
If you could recall from here, this model was broken up into two parts.
So we have let's take this off.
We have this first part which generates this feature maps and then this other parts, which takes in
the feature maps and then outputs.
The specific class of this input image.
But given that when defining our model, this was in a block that is we basically defined this as one,
not as two separate entities like this, we are going to look at how to separate this or better still,
how to create a model which is made up of only the first part and then the model matter of this other
part.
And then one thing you should note is that the way we created this model was a bit different from what
we had seen so far.
So previously, if we get back here with those Pre-trained models, let's get back modeling.
We have pre trained well, let's pick this one chance for learning with this.
You see that we had a backbone defined and then we have the backbone, we have global average pulling
and so on and so forth.
And then when you look at the summary, if you look at the summary, you have all this represented as
one.
So the backbone was or you don't get into the details of what is contained in the backbone.
But now when you look at this order summary, let's get down to Occam.
When you look at this model summary now, you would find that all the details of the backbone are given.
And this is gotten differently because now what we do is the input of our X, which is passing to the
global average.
Pulling is simply this backbone dot output.
So when we say when we specify that you want to get the backbone output, this permits us get a summary
with all the details like this.
And then when it comes to the inputs here, you would say you want to have the backbone inputs.
So let's this input right here is actually useless.
You need to take you could take this off.
So that's it.
So once you specify this and then you have your usual output, then you would get this full summary.
Okay.
So that said, now let's go ahead and create our make this model out of this full model we have here.
So in order to create this last conf layer model, which is essentially this model, here's where we'll
get the output to be this rectified code feature maps, we have this Keras model which takes us input
the Pre-trained model inputs.
So it's quite similar to what we had already here where we just take the inputs to be the backbone inputs.
Now our inputs is the full pre-trained model inputs, but then the outputs take notice.
The output is this last count layer output.
Now what is this last conf layer, the last const, The last conf layer is that layer whose name is
given as top activation.
You'll notice that this is this last conf layer where we have this feature maps, it's eight by eight
by 2048.
And then from here we move on to the global average pulling, then there's one, then two, and then
we have our output.
So this part from your upwards is this initial conv layer, all this convolutional neural network.
And then this here is this classifier unit to the right.
And so now that we have this name of this last conv layer, that is the top activation, we are going
to make use of that to produce our outputs.
You see, last layer is simply the Pre-trained model and we get the layer whose name is last conf layer
name and last name is top activation.
So we understand where we get this top activation from.
So now we have our last layer.
As we've said already, we are now able to have this last Conv layer model or this initial CNN model
year which has as input the image and as output the feature maps.
So let's run this and see what we get.
There we go.
We could, we could do last conv layer model, model and check out the summary.
There you go.
We have this model.
Let's check out the output you see here the model output.
And here is the model's input.
See 256 by two by three.
So we have the the input is the image and the output, the feature maps.
Now we've built this first model or we've created this first model from our overall model.
Let's go ahead and check out on our classifier model.
So for the classifier model, you see we have this inputs.
The input now is going to be this feature maps.
Remember, the output of this was the feature maps and the input of this is the feature maps from here.
Given that we have the inputs to this classifier model, we are going to simply pass this through each
and every layer which makes up the classification part of the Pre-trained model.
So if we get back up here, let's scroll back up.
You'll notice that we had this global average pulling.
We had this dense we had dense one, we had this dense two, which all make up the classifier path.
So let's get back here and then you see this classifier layer name is given here, so we understand
where these names are coming from.
And then we simply go through for every layer name in our classifier layer names.
So let's have this.
So for every layer name in our classifier layer names, we are going to pass the inputs.
This input classifier classifier inputs into this layers.
So we take this input pass into the global average pooling.
Then the output will be x, we take that again, pass this into the dense, then we'll take that again
passing to dense one and then pass into dense two.
And then we have the outputs, which is in this case X.
So our input is the classifier input which is this and our output is the output.
After we pass this input through this different layers, the comb block and the classification unit
designed, the next step will be to compute the partial derivatives of the output with respect to the
features here.
Now, we could compute this partial derivatives of this gradients making use of this gradient method,
which takes in the top class channel and the last conv layer output.
So this year is our top class.
And then here we have our feature maps to obtain this feature mapped As this class conv layer output,
we simply pass the inputs as the image into our last of layer model allows configure model this kind
of layer.
All this conf model right here, take the input pass in here, opt in the feature map.
So use our feature maps and then from the feature maps we pass this to our classifier model.
So we take the classifier, we take the feature maps, pass the classifier model, obtain the predictions
here, then those predictions, and then get the top prediction index.
Now the START prediction index will depend on our example.
Now, given that the person is happy, we expect this index value to be equal one.
So we could we could even print this out.
Let's print this out.
Top prediction index.
There we go.
So we will have this top prediction index and then to obtain the exact score, we are going to simply
pick that score from the predictions.
So here we have the Preds, obviously the prices from this.
So we have the Preds.
And then by specifying top red index or by simply specifying one, what we're saying is want to get
this car when we pass this input into this classifier model and now this car we're talking about here
should be what we had right here.
So it should be this value right here.
So we compete in the partial derivative of the output with respect to the feature maps, and we're making
use of this gradient method to help us do that.
So let's take this off.
You see here we have this year, there we go, we run this and then we obtain this output shape you
see takes the exact shape of the feature maps.
Then once we have this year does this gradients, the next thing we could do is simply obtain the mean
values at every position right here.
So we'll simply make use of this reduce mean while specifying the axis.
So let's run this and print out our pulled Gratz ship.
There we go.
You see we have 2048.
So it's basically this vector now where each position is a mean of a single channel.
Remember, from here we had 2048 channels and each and every one of them was eight by eight.
So.
He should all because he won by eight by eight.
So we had this for each and every channel.
And now we've reduced each and every one of this into a single value.
So this is just a single value.
This is a single value which we have here.
We should see here, essentially.
And so now we have this 2048 dimensional vector, and that's what we've seen here.
So this is essentially this.
Now, the next step will be to take this here and multiply by our feature maps.
To carry out the multiplication.
We'll simply go for Iron Range 2048.
So for each and every one of the different positions we have here, we are going to take the corresponding
last conflict output, which is essentially this feature maps.
So we take the feature map, you see, we specify.
So if zero, then we take in this feature map of this channel and then multiplying by its corresponding
pooled grats, which we've seen already to be the mean of each and every channel we have here for this
gradients.
So that's how we obtain this.
Now note that we specify that one does zero element because this is one by eight by eight by 2048 and
specifying this will leave us with a tensor of shape eight by eight by 2048.
So we get rid of this here.
So that said, we now run this and we obtain our last conflict output.
So it's going to be obviously the same shape you could see.
Check that out.
Last conflict, output shape.
And there we go.
Now, to obtain the heatmap, we are going to sum up the values at different positions for all the channels.
So let's suppose that we have something like this.
Let's say it's, for example, three channels.
Then this position C we have this position, we'll take this, plus this, plus this, and then we will
have this output.
Then we move to the next position, this, plus this, plus this, and then we have this and so on and
so forth.
So that's how we're going to go from this eight by eight by number of channels, output or let's say
2048 output to an eight by eight heatmap.
So this is what we have here.
We're going to sum this and specify that axis is the channel axis.
So we run this already and then now we could visualize our heatmap.
But notice that there's also this railroad right here.
So we have the rail before we visualize the heatmap.
There we go.
Here's our heatmap which we then resize using upon this resize method.
So you see how we get from this heatmap, which was hit by eight to this one, which is 256 by 256.
And the reason why we're doing this is because we want to add this with the image or purpose this on
the image such that we have an output which shows us clearly where all better still regions of the image
where we have the highest contribution to our output, which is that of a happy person.
So the model places that this person is happy and now we know which parts of the image contributed the
most to that prediction.
Now try out with a different image and modifying this slightly so that we could get this image.
Here we find that is this part.
So previously it was a small region.
Now is this reason we you see this wrinkles right here.
We have this zone and this zone which influenced the prediction.
And with this we've just implemented this great camera.
Thank you for getting right up to this point and see you in the next section.